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Introduction 


"Classical'*  aysteas  have  been  uaed  to  provide  tutorial  Instruction 
and  aocratlc  or  supportive  problea  solving*  Tutorial  systcas  ala  to 
diagnose  a  student's  errors  and  then  to  provide  appropriate  raaedlatlon. 
Supportive  problea  solving  systcas  monitor  the  student's  problea  solv¬ 
ing,  and  ala  to  provide  help  and  advice  whenever  requested.  The  sub¬ 
field  of  Intelligent  CAI,  or  ITS  (Intelligent  Tutoring  Systcas),  arose 
because  workers  felt  that  CAI  was  Intrinsically  Halted,  and  In  fact 
incapable  of  providing  highly  adaptive  Instruction^ (Hartley  &  Sleeaan, 
1973;  Sleeaan  &  Brown,  1982).  The  working  hypothesis  of  the  ITS  field 
has  been  that  to  produce  a  highly  successful  tutorial  systea  requires 
^>e  ability  to  Infer  an  accurate  student  aodel,  and  that  It  will  then  be 
relatively  straightforward  to  use  the  aodel  to  direct  a  reaedlal  dialo¬ 
gue.  Reaedlatlon  based  on  a  student  aodel  Is  referred  to  in  this  paper 
as  errcr-speclflc  or  aodel-based  reaedlatlon  (MBR).  MBR  provides  feed¬ 
back  about  specific  error(8)  In  the  student's  procedure  before  reteach- 
Ing  a  correct  strategy.  Its  counterpart,  Reteachlng,  slaply  reteaches 
the  correct  aethod. 

The  PIXIE  systea 

PIXIE  Is  a  data-driven  ITS  shell  (Sleeaan,  1987),  which  atteapts  to 
diagnose  and  then  rcaedlate  student  errors.  Knowledge  about  a  particu¬ 
lar  domain  Is  contained  In  the  knowledge  base.  (At  this  point, 
knowledge  bases  for  linear  algebra,  precedence  In  arithmetic,  negative 
nuabers,  and  fractions  exist;  the  linear  algebra  database  is  the  most 
coaplete. ) 

Each  knowledge  base  Includes: 

-  a  set  of  (correct)  doaain  rules 
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a  aet  of  Incorrect  variant  rulea 


-  aeta  of  examples  that  are  sufficient  to  discriminate  between  the 
models  generated 

The  system  generates  a  aeries  of  models,  that  Include  both  the  correct 
and  all  variants  of  the  incorrect  rules.  In  an  offline  phase.  When  the 
student  works  at  the  terminal,  the  system  merely  has  to  decide  which.  If 
any,  of  the  set  of  predefined  models  fits  the  student's  current  answer. 
For  example,  if  the  student  was  presented  with  the  task  3x  ^  5x  ■  9  and 
thvi  student  gave  x  ■  8/9  as  a  response,  one  explanation  would  be  that 
the  student  had  successfully  added  the  two  x-terms  to  get  8x  ■  9,  but 

then  "Inverted**  the  answer  In  the  final  step.  For  more  details,  see 
Moore  and  Sleenan  (1987).  As  noted  earlier,  the  assumption  made  by  the 
ITS  subfield  has  been  that  it  would  be  relatively  straightforward  to 
remediate  errors  once  they  had  been  accurately  diagnosed,  and  that  MBR 
would  be  more  effective  than  Reteachlng.  A  further  assumption  Is  that 
students  have  (fairly)  stable  mental  models  for  the  task  domain,  and 
that  these  would  be  "used**  consistently  by  the  students. 

The  series  of  studies  reported  in  this  paper  has  attempted  to  emplrl*- 
cally  verify  these  essumptlons,  i.e.,  to  test  the  null  hypothesis  of  no 
difference  between  KBR  and  Reteachlng. 

Review  of  the  literature  on  the  learning  of  algebra 

Diagnosing  algebra  errors  was  investigated  quite  extensively  by  Bucking¬ 
ham  (1933),  and  others;  these  early  investigators  studied  the  types  and 
frequencies  of  errors  found  in  groups  of  students.  Buckingham  (1933) 
suggests  two  uses  for  the  results  from  these  studies: 

First,  the  findings  may  be  utilised  In  planning  the  group  teaching 
procedure.  The  teacher  can  guard  against  the  errors  which  occur 
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with  greatest  frequency  and  peralatency  by  providing  apeclflc 
drills  for  the  class  as  a  whole  in  exaaples  in  which  these  errors 
are  likely  to  occur 

Second,  the  findings  of  the  Investigation  suggest  the  laportance  of 
attention  to  the  individual  needs  of  students.  While,  in  a  general 
way,  certain  types  of  errors  predoainate,  there  Is  no  assurance 
that  any  particular  student  will  aake  errors  either  in  kind  or  fre¬ 
quency  that  correspond  with  those  resulting  froa  this  study. 
Therefore,  the  second  aethod  of  teaching  would  be  to  use  tests  of 
the  type  eaployed  In  this  study  to  dsteraine  the  difficulties  of 
each  student.  Each  student  can  then  be  provided  with  specific 
drill  necessary  to  ovcrcoae  the  errors,  (pp.  101  4  102). 


Several  reviews,  three*  of  which  are  quoted  below,  indicate  that  error- 
specific  reaedlatlon  is  known  to  be  superior  to  reteachlng: 


Brown  and  Burton  foresee  a  tine  when  schools  night  have  s  diagnos¬ 
tic  specialist  who  would  work  with  children  having  special  diffi¬ 
culty  in  nath.  This  diagnostician  would  conduct  in-depth  inter¬ 
views  in  conjunction  with  specific  coaputational  tasks  to  detect 
possible  procedural  errors.  The  intent  would  be  to  gear  instruc¬ 
tion  to  specific  procedural  difficulties  of  an  individual.  (In 
Resnlck  and  Ford,  1981,  p.  88). 


Secondly,  Resnick  (1984)  also  addresses  the  issue  of  what  instructional 
response  is  appropriate  for  errors  that  are  systematic.  Although  her 
investigation  also  centered  on  subtraction  errors,  her  speculations  seem 
appropriate  for  algebra  errors.  Resnlck  (1984)  provides  the  following 
use  of  diagnosing  systematic  errors  for  instruction. 


"....  it  may  prove  useful  to  tailor  practice  to  specific  kinds  of 
buggy  rules,  either  by  choosing  particular  examples  that  are 
matched  to  a  child's  errors  or  by  giving  special  attention  to  the 
parts  of  a  procedure  that  evoke  those  errors."  (p.l3) 


However,  Resnlck  emphasises  that  one  may  derive  the  most  benefit  by 
determining  more  global  misunderstandings  rather  than  remediating  each 
error.  Again,  however,  this  speculation  is  not  supported  by  data. 


*Although  Sleeman  agrees  he  has  made  similar  claims 
orally,  we  are  at  present  unable  to  find  an  appropriate 
written  quotation. 


Thirdly,  M«caab  and  Cumlne  (1986)  dlacuBS  the  laportance  of  showing  the 
student  both  the  correct  procedure  and  pointing  out  the  Incorrect  steps. 
They  state: 


dcaonstratlng  that  there  is  a  flaw  In  a  pupil's  aethod  can  be 
useful  In  situations  where  the  pupil  is  awsre  of  a  correct  pro¬ 
cedure  but  prefers  his  alternative  either  because  he  thought  it  up 
hlaself,  or  It  seems  easier,  or  for  some  other  reason.  In  such 
cases  the  unsound  nature  of  the  pupil's  own  aethod  aay  hsve  to  be 
demonstrated  before  he  will  adopt  a  correct  aethod"  (p.l25). 


The  following  studies,  report  experiaental  results  on  the  effectiveness 
of  various  reaedlal  strategies.  Swan  (1983)  concluded  that  error- 
specific  remediation  which  Involved  an  external  validity  check  is  more 
effective  than  Reteachlng  over  a  period  of  eight  one-hour  lessons. 
Unlike  Swan  (1983),  Bunderson  and  Olsen  (1983),  suggest  that  pointing 
out  an  Incorrect  procedure  is  not  more  effective  than  aerely  reteachlng 
under  some  circumstances: 


Thus  in  the  area  of  subtraction  at  least,  both  error-specific  and 
general  atteapts  at  remediation  for  students  who  had  previously 
been  taught  produced  excellent  results.  Interpreted  In  the  light 
of  error  Instability,  it  seems  that  any  attention  to  correct  a  sub¬ 
traction  error  by  teaching  the  correcting  procedure  will  enable 
students  to  call  upon  their  procedure  and  use  it  correctly  in  a 
posttest  given  at  the  end  of  their  remedial  period.  (Bunderson  and 
Olsen,  1983,  p.ll) 


The  recent  PIXIE  studies  (Kelly,  Martlnak,  &  Sleeman,  1987;  Kartlnak, 
Schneider,  &  Sleeman,  1987)  cast  serious  doubt  on  the  need  for  a  diag¬ 
nostic  specialist.  In  addition,  Putnam  (1987)  and  Kelly  and  Sleeman 
(1986)  found  that  teachers  generally  do  not  adopt  the  role  of  a  diagnos¬ 
tician,  even  when  in  a  tutorial  situation.  Diagnosing  errors  tends  not 
to  be  the  teacher's  primary  goal.  Putnam  suggested  that  teachers  in 
their  remediation  followed  a  "currlculum-based-scrlpt".  However,  Putnam 
did  not  compare  the  effectiveness  of  tutoring  based  on  a  curriculum 
script  with,  say,  an  MBR  approach.  Given  the  conflicting  results 


reported  above,  aorc  raaearch  Ic  obviously  needed  to  ayateaatlcally  com¬ 
pare  error-specific  renedlatlon  (or  aodel-based  renedlatlon  -  MBR)  with 
Keteachlng. 


The  rest  of  the  paper  reports  on  the  developaent  of  the  KPIXIE  aysten 
and  on  the  detailed  aeries  of  cxperiaents  carried  out  to  address  these 
iasues. 

!•  Developaent  of  the  Coaputer-based  PIXIE 
A.  Student  Protocols;  PIXIE 'a  Diagnostic  Systea 

Protocols  of  students  working  various  types  of  aatheaatlcs  tasks  were 
analysed  to  determine  the  types  of  errors  aade,  and  these  were  used  to 
create  a  knowledge-base  for  PIXIE.  The  domains  Investigated  by  the 
PIXIE  Projecf  include:  linear  algebra,  fractions,  precedence  In  arith- 
aetlc,  and  negative  numbers. 


II.  Pilot  Studies 

A.  Revisions  to  PIXIE  (Diagnostic  System) : 

In  1984  and  early  1985,  PIXIE  (only  a  diagnostic  system  at  this  point) 
was  field  tested  in  several  schools.  Overall,  the  reaction  to  PIXIE  was 
favourable;  however,  many  atudents  did  not  type  complete  algebraic  equa¬ 
tions  at  each  step  (a  requirement  of  the  first  version  of  PIXIE).  PIXIE 
was  subsequently  revised  to  include  an  alternative  "Interface"  that 
requires  only  the  final  response,  and  not  intermediate  steps,  be  valid 
algebraic  expressions.  During  these  field  trials,  mal-rules  that  were 
not  ancountered  in  earlier  work  were  noted  and  subsequently  added  to 
pixie's  rule-base. 
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B.  Teacher  Protocols ;  PlXIE'a  Reaedlal  aubsystem. 

With  the  Intention  of  building  a  reaedlal  aub-system  in  PIXIE,  aeveral 
atudlea  of  teachera  perforalng  tutoring  were  undertaken  (Kelly  &  Slee- 
man,  1986).  In  the  flrat  atudy,  four  experienced  teachera  were  eaked  to 
dlagnoae  and  auggeat  remediation  for  a  aerlea  of  cquatlon-anawer  pairs, 
which  were  based  on  actual  student  protocols.  In  the  second  atudy,  an 
axperlenced  teacher  remediated  eight  students,  based  on  diagnostic 
information  provided  by  the  PIXIE  aystem.  Kesults  from  this  series  of 
studies  Indicated  that: 

a)  Teachers  frequently  do  NOT  make  a  very  detailed  diagnosis. 

b)  Remediation  Is  by  and  large  very  procedurally  orientated  -  few 
teachers  stress  conceptual  understanding. 

c)  Remediation  frequently  takes  the  form  of  reteachlng. 

C.  Design  of  RPIXIE 

On  the  basis  of  teacher  protocols,  an  embryonic  remedial  system  was 
Implemented.  The  remedial  system  provided  two  basic  types  of  remedia¬ 
tion:  Model  Based  Remediation  (MBR)  and  Reteachlng.  MBR  points  out  and 
labels  as  Incorrect  specific  error(s)  made  by  the  student  before 
presenting  a  correct  method  for  solving  the  task.  Reteachlng  merely 
presents  a  correct  method  for  solving  a  task  without  pinpointing 
error(s).  The  following  are  examples  of  MBR  and  Reteachlng: 


An  Example  of  a  Student  Interaction:  HBR 


(This  protocol  is  slightly  edited) 


Solve  (f  ■  Finished,  q  •  Quit  early) 

7x  -  2  *  2 
S:  7x  -  A 
S:  f 

Simplify  further  7x  •  A 
S;  X  "  A 
S:  f 

Thank  you, 

but  you  didn't  get  that  one  right. 

We  will  now  look  at  that  Cask  again. 

Here  we  are  considering  Che  cask  7x  ■  2  *  2 

Did  you  work  out  Che  Cask  something  like  this?: 

7x  -  2  *  2 
7x  -  A 
7x/7  -  A 
X  ■  A 

Please  type  YES  or  NO:  y  (Student  replies) 

It  seems  that  you  changed  7x  ■  A  to  something  like  7x/7  ■  A 
This  Is  WRONG. 

Do  NOT  only  divide  ONE  side  by  the  coefficient. 

You  should  divide  both  sides  of  the  equation  by  the  coefficient  to 
change 

7x  -  A  to  7x/7  -  A/7 


Now  let  me  show  you  how  you  SHOULD  have  wprked  clie  whole  task 


7x  ~  2  *  2  ■ 

multiply  2  numbers 

7x  -  A  ' 

divide  both  sides  of  the  equation  by 

cient 

7x/7  -  A/7 

divide 

1 

K 

Finished 

coef fi- 


An  Example  of  •  Student  Interaction:  Reteachlng 

(Same  taak;  again  the  protocol  has  been  edited  slightly  and  the  Initial 
student  interaction  has  been  omitted.) 

Thank  you, 

but  you  didn't  get  that  one  right. 

Ve  w'.j,  now  look  at  that  task  again. 

How  let  me  show  you  how  you  SHOULD  have  worked  the  whole  task 

7x  ■  2  *  2  multiply  2  numbers 

7x  ■  4  divide  both  aides  of  the  equation  by  the  coefficient 

7x/7  ■  4/7  divide 

X  >  4/7  Finished 

0.  Kevlslons  to  RPIXIE  (the  diagnostic  and  remedial  systems)  as  £ 
result  of  £  pilot  study; 

RPIXIE  (the  remedial  version  of  PIXIE)  was  taken  to  several  schools  to 
Investigate  whether  students  could  use  It  productively.  The  Initial 
version  of  RPIXIE  required  the  student  to  work  six  tasks  of  the  same 
format  before^  receiving  feedback  about  the  correctness  of  any  item.  If 
at  least  4  of  the  6  tasks  were  worked  consistently  using  the  same  mal*' 
rule(s),  the  student  received  KBR  on  those  tasks;  for  the  other  tasks  In 
the  set,  the  student  received  Reteachlng.  If  tasks  had  been  incorrectly 
worked,  but  with  less  consistency,  then  the  student  received  Reteachlng 
for  all  6  Items.  However,  if  MBR  was  chosen,  the  students  were  required 
to  recall  how  they  worked  each  task;  students  often  could  not  remember. 
On  some  occasions,  the  MBR  trace  was  interpreted  as  the  correct  pro¬ 
cedure  by  some  students.  Results  from  these  field  studies  suggested  the 
following  revisions  to  RPIXIE: 

a)  Require  the  student  to  work  fewer  items  of  the  same  format,  and 
provide  a  mode  in  which  it  is  necessary  to  work  only  a  single  task 
before  feedback  and.  If  necessary,  remediation  Is  given. 

b)  Shorten  the  text  students  are  required  to  read.  Provide  deflnl- 
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tlons  for  techolcal  tens  (e.g.,  coefficient). 

c)  Highlight  the  HBR  trace  ao  that  It  la  aore  obvious  to  the  student 
that  it  repreaents  an  incorrect  aethod. 

d)  Provide  two  algebraic  cancelling  aethods:  one  approach  should  use 
'aove  and  change  the  sign**,  the  other  "cancel  by  doing  the  aane 
thing  to  both  sides*’. 

e)  Allow  the  student  to  select  from  a  series  of  reaedlal  aodels.  Ini¬ 
tially,  RPIXIE  reaedlated  only  when  a  single  aodel  fitted  the 
student's  response;  the  issue  of  which  of  several  aodels  to  present 
to  the  student  has  subsequently  been  addressed  (Moore  6  Sleeman, 
1987). 

These  changes,  with  the  exception  of  (c),  were  impletnented. 

FORMAL  EXPERIMENTS 

When  a  workable  diagnostic  and  remedial  system  was  in  place,  a  series  of 
studies  to  investigate  its  effectiveness  were  carried  out.  PIXIE  was 
developed  under  the  assumption  that  diagnosing  a  student's  error(s)  and 
specifically  remediating  the  crror(s)  before  showing  the  correct  pro¬ 
cedure  (KBR)  was  educationally  more  beneficial  than  simply  showing  the 
student  the  correct  procedure  (Reteschlng) .  The  series  of  studies  dis¬ 
cussed  below  were  carried  out  to  test  this  assumption. 

Six  studies  were  conducted:  three  studies  used  the  computer  as  a  tutor 
and  the  other  three  studies  used  huuns  ss  tutors.  All  six  studies  were 
based  on  a  pretest  -  intervention  -  posttest  design.  Some  of  the  stu¬ 
dies  reported  below  hsve  fewer  students  than  we  had  hoped  for  because 
only  a  subset  of  the  pretested  students  would  need  tutoring,  or  in  the 
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case  of  s  later  study,  only  a  subset  had  stable  errors.  When  confronted 
with  the  number  of  qualifying  students,  we  had  to  decide,  for  example, 
whether  to  Increase  the  number  of  students  In  each  condition  at  the 
expense  of  the  control  group.  This  decision  was  guided  by  the  major 
hypothesis  being  tested. 

The  studies  are  eummarlxed  In  (approximate)  chronological  order  so  that 
the  reader  may  gain  a  better  understanding  of  the  Issues  as  they  arose. 

I.  Initial  Computer  Based  Remediation; 

In  this  first  study  (California,  spring  1986),  the  effectiveness  of  dif¬ 
ferent  forms  of  remediation  was  investigated  by  using  an  intelligent 
tutoring  system  (RPIXIE).  The  null  hypotheslt  was  that  there  would  be 
no  difference  In  performance  among  three  conditions  of  tutoring:  MBR, 
Reteschlng,  and  Evaluation  (simple  knowledge  of  performance). 

RPlXIE's  Algebra  Pats  Base 

RPlXlE's  algebra  data  base  consists  of  17  algebra  task-sets  that  vary  In 
difficulty  from  Items  of  the  form  ax~b  to  items  of  the  form 
ax-b*c(dx+e) .  In  this  study,  three  of  RFIXlE's  presentation  modes 
(forms  of  remediation)  were  used: 

(1)  HER:  remediation  In  which  students  were  shown  whst  they  had  done 
wrong,  and  then  presented  with  s  correct  solutlon-trar ' . 

(2)  Reteschlng:  remediation  In  idtlch  students  were  merely  Informed 
whether  their  responses  were  correct,  sod  then  presented  with  a 
correct  solution-trace. 

(3)  Evaluation:  remediation  In  which  students  were  Informed  only 
whether  items  were  answered  correctly  or  Incorrectly,  with  no 


correct  solution-trace  shovm. 

Students 

IVo  cxperlaents  were  run,  both  with  Identical  procedures.  The  first 
experiment  Involved  15  ninth-  and  tenth-grade  algebra  students  from  a 
high-achieving  mathematics  class  (standardised  mathematics  achievement 
test  scores  were  at  or  above  tenth-grade  level).  In  the  second  experi¬ 
ment  ,  24  students  from  a  low-achieving  algebra  class  were  tested  (test 
scores  were  below  the  eighth-grade  level).  The  students  came  from  a 
high  school  In  the  San  Francisco  area  (School  C). 

Materials 

Prior  to  the  experiment,  data  were  collected  on  the  mathe  >tlcs  section 
of  the  Standardised  Test  of  Educational  Progress  (STEP),  and  on  the 
Mathematics  Attitude  Inventory  (MAI)  -  a  survey  containing  six  subscales 
relating  specifically  to  mathematics:  attitude  towards  the  teacher, 
motivation,  anxiety,  enjoyment,  confidence,  and  value  of  mathematics. 

Materials  also  Included  a  17-Item  pretest  and  17-ltem  posttest.  Both 
tests  contained  algebra  tasks  that  were  similar  to  those  worked  Indivi¬ 
dually  with  RPIXIE.  Tasks  on  the  pretest  and  posttest  were  matched  for 
difficulty,  by  using  the  sane  templates  (e.g.,  ax4-b*c),  to  generate 
tasks  for  both  tests.  A  four-item  questionnaire  reviewing  RPIXIE  was 
also  developed. 

Procedure 

Students  In  the  classes  took  a  group  pretest  am'  the  MAI  during  the  reg¬ 
ular  class  period.  The  week  following  the  tests,  students  were  randomly 
assigned  to  condition:  MBR,  Reteachlng,  or  Evaluation.  Very  little 


reaearcher/student  Interaction  occurred  while  the  atudent  individually 
worked  with  RPIXIE.  If  the  atudent  did  not  underatand  the  inatructions, 
they  were  explained  by  the  reaearcher.  All  atudenta  began  at  the  eaal- 
eat  level  (Taak  Set  1)  and  worked  until  all  17  taak-seta  were  eoapleted 
or  45  minutea  had  elapaed  ^  whichever  waa  aooner.  Studenta  were  then 
given  5  ainutea  to  anawec  the  four-itea  queationnaire.  After  all  Che 
atudenta  had  interacted  with  RPIXIE,  a  group  poatteat  waa  adainiatered 
during  the  regular  aatheaatica  period. 

Reaulta 

Effectiveness  of  feedback  was  aeasured  by  the  posttest,  by  the  nuaber  of 
iceas  correctly  answered  on  RPIXIE,  and  by  Che  percentage  of  iteas 
atteapted  on  RPIXIE  that  were  answered  correctly.  See  Table  la  and 
Table  lb  for  aean  scores  by  condition  for  each  group. 


Table  la.  High  Achieving  Group 

Mean  niUDber 
of  Items 
worked 


Preteat* 

Posttest* 

on  RPIXIE 
per  student 

X  correct 
on  RPIXIE 

MBR 

13.00 

(5.35) 

13.20 

(6.63) 

56.00 

(3.00) 

81 

Eeteachlng 

14.60 

(1.50) 

53.20 

(2.82) 

91 

Evaluation 

15.60 

(1.34) 

14.40 

(2.30) 

56.20 

(7.82) 

66 

*  Maximum  ■ 

17,  etandard  deviations 

are  shown  in  parentheses.  N  *  15. 

Table  lb.  Low  Achieving  Group 

Pretest* 

Posttest* 

Mean  number 
of  items 
worked 
on  RPIXIE 
per  student 

X  correct 
on  RPIXIE 

MBR 

9.25 

(4.03) 

10.63 

(3.30) 

51.63 

(4.31) 

84 

Bcteach 

10.29 

(3.35) 

12.13 

(1.55) 

51.75 

(8.00) 

86 

Evaluation  8.14 
(2.73) 


10.00 

(3.02) 


49.75 

(14.68) 


67 


Results  by  Condition 


Mo  significant  differences  saong  the  conditions  was  found  for  the  high- 
achieving  group.  For  the  low-achieving  group  the  results  were  similar. 
We  found  no  algnlflcant  differences  by  condition  (Hotelling's  t(14)  ■ 
1.77,  p<.ll),  although  the  atudents  in  the  Evaluation  condition  appear 
to  have  worked  fewer  items  correctly  on  the  computer.  This  may  be 
because  of  poor  motivation  due  to  not  receiving  remedial  aupport  for 
wrong  answers.  (Mote:  the  apparent  heterogeneity  of  variance  for  the 
mean  number  of  items  worked  on  RPIXIE  is  not  of  concern,  because  the 
differences  between  the  means  do  not  appear  to  be  of  practical  conse¬ 
quence.) 

RPIXIE'a  diagnostic  power 

The  data  were  analysed  to  determine  the  percentage  of  errors  that  RPIXIE 
diagnosed.  Specifically,  we  asked:  1)  Did  PIXIE  have  a  model  that  the 
researchers  believed  explained  the  student's  answer,  and  2)  Did  the  stu¬ 
dent  agree  that  RPIXIE's  model  was  acceptable?  Under  the  first  cri¬ 
terion,  RPIXIE  diagnosed  approximately  33Z  of  the  errors  made  in  the  HBR 
condition;  the  students  "approved”  91Z  of  these,  resulting  in  30Z  of 
errors  being  followed  by  MBR. 

Discussion 

The  finding  of  no  significant  differences  between  the  MBR  and  Reteachlng 
conditions  may  be  due  to  RPIXIE's  low  diagnostic  rate  (70X  of  Che  atu¬ 
dents'  errors  in  the  MBR  condition  were  followed  with  ReCesching).  This 
rendered  the  two  groups  more  similar  than  dissimilar  because  the  default 
is  Reteachlng.  This  low  rate  of  diagnosis  arose  because  of  the  limited 
number  of  mal-rules  used  in  RPIXIE,  which  in  turn  arose  for  a  variety  of 
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technical  rcasona  that  have  now  been  ■ubatantlally  overcome. 


II.  Human  Remediation;  A  Study  to  Compare  HBR  and  Reteachlng 

The  above  atudy  led  us  to  believe  that  the  Issues  of  diagnosis  and 
remediation  were  much  more  aubtle  than  Initially  suspected,  ve  decided 
to  replicate  the  atudy  using  human  tutors.  The  first  such  experiment 
was  carried  out  in  the  Autumn  of  1986  at  an  Aberdeen  school,  School  L. 
The  null  hypothesis  was  that  there  would  be  no  difference  in  performance 
between  two  conditions  of  tutoring:  KBR  and  Reteachlng. 

Subjects :  Students  from  two  2nd*  and  two  3rd-year  mathematics  classes  In 
School  L  participated  In  the  study.  Average  ages  of  the  students  were 
13  years  4  months  and  14  years  6  months,  respectively.  On  the  basis  of 
an  analysis  of  each  student's  pretest,  a  subset  of  44  students  were 
selected  for  Individual  tutoring. 

Materials:  Materials  Included  jt  20-ltea  pretest.  £  20-1 ten  posttest . 
and  scripts  for  remediation.  As  In  the  previous  study  the  same  tem¬ 
plates  were  used  to  generate  the  items  on  both  tests.  See  Appendix  A 
for  the  pretest  and  posttest  items. 

Tutoring  scripts  based  on  RPIXIE's  approach  to  remediation  were 
developed.  Separate  scripts  were  written  for  MBR  and  Reteachlng.  (See 
Appendix  B  for  sample  scripts.)  The  MBR  script  directs  the  tutor  to 
point  out  to  the  student  each  error  made  and  to  explain  It  before  the 
correct  procedure  Is  retaught.  The  Reteachlng  script  merely  directs  the 
tutor  to  reteach  the  procedure.  Items  to  be  worked  during  tutoring  were 
written  prior  to  the  sessions  so  as  to  ensure  uniformity  across  the 
several  tutors,  l.e.,  students  making  an  error  In  a  given  Item  were 
given  the  same  remedial  tasks  Irrespective  of  the  tutor.  Each  tutor  was 
trained  with  these  scripts.  (See  Appendix  C  for  the  Items  used  at  each 
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level  of  tutoring-) 


Procedure;  All  etudente  In  the  study  took  s  iO-alnute  group  pretest  dur¬ 
ing  their  regularly  scheduled  Bsthematics  class.  A  subset  of  these  stu¬ 
dents  were  rendowly  assigned  to  KBR  or  Reteschlng.  Each  student  was 
Individually  tutored  for  approxiaately  35  minutes.  Tutoring  occurred 
the  week  following  the  pretest,  and  all  sessions  were  completed  within  a 
week.  All  tutoring  sessions  were  audio-taped.  Tutoring  consisted  of 
having  the  student  first  rework  an  Item  marked  as  Incorrect  on  the 
pretest.  If  the  item  was  again  worked  incorrectly,  remediation 
appropriate  to  the  condition  was  given  and  the  student  worked  at  most 
two  practice  items  of  the  same  type.  This  procedure  was  repeated  for 
each  item  scored  as  Incorrect  on  the  pretest. 

The  week  following  tutoring,  a  group  posttest  was  given  to  the  students 
during  a  regular  mathematics  class.  A  delayed  posttest,  identical  to 
the  isBBediate  posttest,  was  given  approximately  two  months  after  the 
first  posttest. 


Results : 
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Analyses  of  the  data  are  based  on  scores  from  36  of  these  students 
(because  6  were  absent  from  school  for  the  tutoring  or  the  posttest). 
Posttest  scores  were  taken  as  a  aeasurement  of  the  effectiveness  of 
tutoring.  Table  2  presents  the  aean  scores  by  con. '.tlon;  standard  devi¬ 
ations  are  given  in  brackets. 


Table  2*  Wean  Scorea*  by  Condition 


Condition 

Pretest 

Posttest  1 

Posttest  2 

MBR 

12.53 

14.32 

12.24 

(3.86) 

(3.37) 

(5.62) 

Reteachlng 

11.63 

14.42 

12.76 

(3.10) 

(3.64) 

(4.44) 

*Haxlmum  ■  20;  N  ■  3 


Although  there  waa  a  algnlf leant  overall  aean  difference  between  the 
pre-  and  poattaata  for  both  groupa  >  4.20,  £  <.001,  there  was  no 
algnlficant  difference  by  condition:  ~  *09,  £  >.92.  The  overall 
■ean  acorea  for  the  delayed  poetteat  (two  aontha  from  the  flrat  poet- 
teat)  were  alao  not  algnlficant  by  condition  t^^  *  .30,  p>.70,  and  had 
reverted  to  preteata  levela. 

Further  analyaea  of  the  data  uaing  only  atudente  who  were  poor  on  the 
preteat  (l.e.,  thoae  acorlng  13  or  leaa)  ahowed  no  algnlficant  dlffer- 
encee  by  condition,  t,cj  ■  0.43,  £  >.66.  Errora  were  claaalfled  Into 
aeveral  levela  of  aeverlty,  but  again  no  correlation  vaa  found  between 
the  nuaber  of  'aevere”  errora  Bade  and  the  tutoring  condition. 

Dlacuaslon 

Thia  experiaent  confirned  the  previous  computer  study  results  and  showed 
that  KBR  and  Reteachlng  are  very  comparable,  even  with  low-scoring  stu¬ 
dents.  We  Interpreted  this  to  imply  that  the  form  of  MBR  used  was  not 
effectively  communicetlng  with  the  student.  Study  III  attempted  to  make 
MBR  more  effective. 


Subsequent  studies  do  not  attempt  to  categorise  the  severity  of  errors 
as  we  did  here,  because  of  the  problems  Involved  with  the  classifica¬ 
tion.  Definitional  difficulties  were  encountered  because  we  did  not 
have  information  pertaining  to  why  students  made  particular  errors.  For 


example,  ie  an  Inverted  division  (l.e.,  ex  ■  b  ■>  x  ■  e/b)  e  careless 
error,  an  algebrlc  misunderstanding,  or  a  severe  misunderstanding  of 
fractions? 
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III.  Human  Remediation;  A  study  to  compare  variants  of  MBR  and  Reteach- 


Ing 

A  study  was  run  to  investigate  two  factors  that  may  have  influenced  the 
results  from  the  above  study:  namely  presumed  lack  of  cognitive  disso¬ 
nance  and  lack  of  cognitive  engagement  on  the  part  of  the  students.  It 

was  hypothesised  that  no  differences  between  the  conditions  in  the  pre¬ 

vious  study  had  arisen  because: 

a)  The  atudents  had  no  reason  to  accept  the  tutor's  nethod  of  doing 

algebra  as  better  than  their  own.  Macnab  and  Cuomlne  (1986)  dis¬ 

cuss  the  importance  of  demonstrating  to  the  student  the  unsound 
nature  of  the  pupil's  incorrect  nethod,  i.e.,  create  'cognitive 
dissonance'  (CO).  In  the  present  study,  we  attempted  to  instil 
cognitive  dissonance  by  having  students  check  their  answer  by  sub¬ 
stituting  it  back  into  the  equation  to  see  if  both  sides  of  the 
aquation  balanced. 

b)  The  students  were  not  sufficiently  involved  with  their  learning 
(l.e.,  they  were  passive  listeners  to  the  tutor's  Instructions). 
By  having  students  verbally  repeat  the  correct  procedure  back  to 
the  tutor,  we  hoped  to  engage  them  more  in  their  learning. 

Procedure;  Students  from  two  2nd-and  one  3rd-year  mathematics  classes 
(average  age  13  years  6  months  and  14  years  8  months,  respectively)  from 
a  different  Aberdeen  secondary  school  (School  P)  took  a  20-ltem  pretest 
in  algebra  aquation  solving.  Anyone  scoring  602  or  better  on  the 
pretest  was  not  seen  for  tutoring.  Thus,  a  subset  of  48  students  were 
randomly  assigned  to  one  of  four  conditions: 
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b)  Reteaching 


c)  MBR  vlth  cognitive  dlaaonance  (MBR  +  CD) 

d)  MBR  with  cognitive  cogageaent  (MBR  -4-  CE) 

Given  that  there  were  a  Halted  nuaber  of  students  to  work  with,  the 
nuaber  of  students  per  condition  was  increased  at  the  expense  of  a 
pretest /posttest-only,  Reteach  CD^  and  Reteach  CE  conditions,  in 
order  to  better  test  the  differences  aaong  the  above  four  conditions. 
(Note  that  the  MBR  and  Reteachlng  conditions  used  in  the  previous  study 
were  designed  into  this  study,  so  that  the  previous  study  could  be 
replicated.)  See  Appendix  D  for  samples  of  the  scripts  used  in  each  of 
the  four  conditions. 

Each  student  was  individually  tutored  for  approximately  35  minutes.  All 
tutoring  sessions  were  audiotaped.  Tutoring  consisted  of  having  the 
student  first  rework  sn  item  alssed  on  the  pretest.  If  the  item  was 
again  worked  incorrectly,  the  student  received  reaedlatlon  appropriate 
for  the  treataent  condition  and  worked  at  aost  two  more  practice  items 
of  the  same  format.  To  discourage  a  student  in  the  Reteachlng  condition 
from  making  coaparlsons  between  the  Incorrect  and  correct  procedure,  the 
student's  workings  of  the  incorrect  procedures  were  taken  away  before 
the  correct  procedure  was  taught.  After  each  of  the  students  were 
tutored,  group  postteste  were  given  to  all  students  in  the  classes 
involved. 

Results; 

Both  quantitative  and  qualitative  analyses  were  performed  on  the  data. 


Results  froa  an  AMOVA  Indicated  no  algnificant  differences  among  condl 


tlons  on  the  pretest:  “  .626,  £  >.50,  (see  Table  3  for  means). 

Table  3.  Pretest,  Posttest,  and  Delayed  Posttest  mean  scores  by  conditio: 

Condition 

Mean 

Pretest 

Scores* 

Mean 

Posttest 

Scores* 

Delayed 

Posttest 

Scores* 

Reteach 

12.08 

15.20 

15.64 

(1.98) 

(2.92) 

(3.07) 

MBR 

11.91 

15.80 

14.73 

(3.33) 

(3.68) 

(2.05) 

KBR^CD 

14.00 

(2.63) 

KBR+CE 

12.91 

15.90 

13.00 

(2.81) 

(3.33) 

(1.80) 

*  Maximum  score  •  20. 

N  >  46;  standard  deviations  are  given  in  brackets 

The  overall 

mean  for 

the  posttest  acore  was  algnlflcantly  higher  than 

the  overall 

mean  for 

the  pretest  (t 

•  5.83,  p<.001)  showing  a  general 

pre-  to  posttest  gain 

.  An  ANOVA  on 

posttest  scores  showed  no  slgnifl- 

cant  differences  among  the  conditions:  ”  *797,  p>.50.  An  ANOVA  on 
the  delayed  posttest  scores  also  showed  no  differences  by  condition:  F; 

-  11.81  p>.10. 

Because  the  above  analyses  showed  no  differences  among  the  groups, 
errors  were  reclassified  as  algebraic  or  non-algebralc  (see  Appendix  E 
for  examples).  The  mean  number  of  algebraic  errors  for  each  condition 
on  the  (first)  posttest  were:  Reteach:  2.00,  MBR:  2.09,  MBR-K:D:  3, 
HBR4CE:  2.73,  indicating  no  major  differences  among  the  conditions. 

Following  these  results,  we  hypothesized  that  a  significant  number  of 
student  errors  might  be  unstable;  which  we  Investigated  by  tracing  a 
given  error  for  a  given  task  from  pretest,  to  tutoring,  to  posttest. 
(In  retrospect,  we  believe  that  this  is  a  stringent  stability  cri¬ 
terion.)  This  analysis  showed  that  approximately  60Z  of  the  errors  from 
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Che  pretest  across  conditions  did  not  occur  on  the  sane  Iteas  during 
tutoring.  However,  intenedlate  tutoring  on  items  may  have  lowered  this 
stability  measure  since  the  arror  on  subsequent  items  may  have  been 
corrected  as  a  result  of  tutoring  on  previous  items.  If  stable  errors 
are  reclassified  to  Include  those  tutored  in  an  earlier  part  of  the  ses¬ 
sion,  the  average  percentage  of  stable  errors  during  tutoring  Increases 
to  26Z. 


An  analogous  retrospective  analysis  of  the  data  from  the  study  II  at 
School  L  was  carried  out,  which  confirmed  these  results. 


Discussion 


Three  analyses  were  applied  to  this  study,  each  of  wh  .‘h  was  logically 
driven  by  the  earlier  ones.  Analyses  1  and  2  did  not  find  differential 
effects  among  conditions,  in  spite  of  Che  modifications  to  the  basic  MBR 
treatment.  (It  should  be  noted,  however,  that  the  MBR  -f  CD  condition 
could  not  be  properly  tested  with  this  sample,  because  the  process  of 
subscltuclon  and  verification  was  new  to  these  students.)  Analysis  3 
suggested  that  the  phenomena  of  student  errors  is  more  complex  than  we 
had  anticipated  and  pointed  strongly  to  the  instability  of  a  high  pro¬ 
portion  of  student  errors.  No  further  conclusions  were  drawn  from  this 
experiment  as  it  was  not  designed  to  investigate  stability;  stability 
became  the  focus  of  study  VI. 


IV.  RPIXIE-Based  Remediation 


The  reader  will  recall  that  the  main  conclusion  from  study  I  (the  previ¬ 
ous  computer-based  study)  was  that  MBR  and  Reteachlng  were  comparable, 
and  this  in  turn  was  attributed  to  the  poor  diagnostic  capability  of 
RPIXIE.  Subsequently,  additional  mal-rules  were  added  to  the  algebra 
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knowledge  base,  and  RPIXIE  has  been  enhanced  so  that  It  can  offer  diag¬ 
noses  in  the  cases  in  which  nultiple  modes  explain  the  student's  error. 
Given  that  the  above  studies  with  human-tutors  have  shown  that  MBR  and 
Reteaching  are  highly  comparable,  the  foci  of  this  study  were: 

a)  Has  the  diagnostic  capability  of  RPIXIE  improved  with  the  addition  of 
aal-rules,  and  b)  Uould  differences  arise  between  conditions  when  the 
computer  (namely  RPIXIE)  provided  the  remediation,  which  perhaps  might 
not  emerge  with  a  human  tutor? 

Procedure;  Forty-one  students  from  2nd-and-3rd-year  classes  (in  School 
P)  were  randomly  assigned  to  condition:  MBR,  Reteachlng,  or  Control. 
Students  in  the  control  condition  did  not  Interact  with  RPIXIE.  All 
students  took  a  20-ltem  pretest  during  their  regular  maths  class.  The 
week  following  the  pretest,  students  in  the  treatment  groups  Individu¬ 
ally  interacted  with  RPIXIE.  Very  little  researcher/student  interaction 
occurred  while  the  student  worked  with  RPIXIE.  If  the  student  did  not 
understand  the  instructions,  they  were  explained  by  the  researcher.  All 
students  began  on  RPIXIE  at  the  easiest  level  (Task  Set  1)  and  worked 
until  all  task-sets  were  completed  or  A5  minutes  had  elapsed,  whichever 
was  the  sooner.  Students  were  then  given  5  minutes  to  answer  a  4-ltem 
questionnaire  pertaining  to  their  experiences  with  the  program.  After 
all  the  students  had  interacted  with  RPIXIE,  a  group  po'test  was  admin¬ 
istered  during  the  regular  mathematics  period. 


Kesults ; 


The  data  were  first  analysed  to  detervlne  the  percentage  of  errors  that 
RPIXIE  diagnosed.  As  before,  diagnosis  was  defined  In  two  ways:  a)  Did 
PIXIE  have  a  aodel  that  the  researchers  believed  explained  Che  student's 
answer,  and  b)  Old  the  student  agree  that  RPIXIE's  aodel  was  accept¬ 
able?* 


Under  the  first  criterion,  RPIXIE  diagnosed  approxlnately  60Z  of  the 
errors  made  by  students  In  the  MBR  condition;  Che  student's  ‘'approved'' 
29Z  of  these,  resulting  In  17Z  of  errors  being  followed  by  MBR.  Studies 
II  and  III  reported  here  suggest  that  approximately  13Z  of  the  errors 
udents  make  while  solving  algebra  Items  are  computational.  When  one 
takes  this  Into  account,  RPIXIE's  matching  of  non-arlthmetlc  errors 
Increases  on  the  two  measures  given  earlier  to  69  OX  (60  out  of  87)  and 
19. 5Z  (17  out  of  87)  respectively**. 


*  RPIXIE  allows  the  student  to  disagree  with  the  in¬ 
correct  procedure  produced  by  Che  inferred  model.  Al¬ 
lowing  the  student  to  disagree  with  the  model  prevents 
students  from  seeing  remediation  that  does  not  reflect 
the  particular  Incorrect  procedure  used.  A  student  may 
disagree  with  RPIXIE's  model  because  the  trace  contains 
steps  the  student  did  not  do.  For  example,  the  model 
may  show  the  following: 

3x  •  9 
3x/3  -  3/9 
X  -  3/9 
X  -  1/3 

The  student  may  reject  this  trace  because  the  step  3x/3 

•  3/9  was  not  typed  by  him. 

**  The  analysis  of  study  VI  showed  that  unstable,  undi¬ 
agnosed  and  computational  errors  account  for  approxi¬ 
mately  15Z  of  errors.  (See  the  discussion  of  study  VI 
for  the  definition  of  stable  errors  and  for  further  de¬ 
tails.)  However,  that  may  not  be  the  appropriate  figure 
for  computer-based  studies;  there  are  Indications  that 
the  error  rate  nay  be  higher,  (Sleeman,  1982). 


Table  U  -  Pretest  and  Posttcst  Scores 


Condition 


Model^based 


Control 


Mean 
Pretest 
Score 


5.00 

2.08) 


Mean  Number 
Mean  of  Items 

Posttest  worked  on  X 

Score  RPIXIE  correct 


6.13 

3 

2.83) 

The  analyses  In  Table  4  ahowed  no  differences  on  Che  posttest  by  condi¬ 
tion  P><30):  standard  deviations  are  given  In  brackets.  A 
differencial  effect  for  tutoring  may  not  have  been  found  because  stu¬ 
dents  worklrg  with  RFIXIE  did  not  In  fact  receive  much  tutoring,  essen¬ 
tially  only  4  tasks  (102  of  40)  on  average  were  worked  incorrectly  on 
both  conditions. 

Discussion 

KPIXlE's  diagnostic  capability  has  been  improved  since  study  I.  Addi¬ 
tionally,  a  capability  to  handle  multiple  models  was  added;  however,  as 
Table  4  shows,  the  students  incorrectly  worked  only  a  small  number  of 
tasks.  Thus,  there  was  relatively  little  opportunity  for  RPZXIE  to  pro¬ 
vide  either  HBR  or  Reteachlng  to  this  sample,  which  probably  helps 
explain  the  lack  of  difference  among  the  three  conditions  on  Che  post- 
test.  Consequently,  the  hypothesis  chat  MBR  with  RPIXIE  might  be  better 
than  MBR  with  humans,  is  left  untested. 

The  notable  change  between  study  I  and  this  study,  is  the  reduction  from 
902  to  292  In  the  students'  ’’approval"  of  the  MBR  traces.  This  can  be 
explained  by  a  difference  In  population,  but  more  particularly  by 


"v.V, 


changes  In  the  knowledge-base  which  resulted  in  highly  redundant  and 
longer  traces.  Moore  and  Sleeman  (1987)  discuss  cnhancceents  to  address 
this  latter  concern. 


mimsm 


V.  A  Third  Study  Using  RPIXIE 

An  RPIXIE  study,  analogous  to  study  IV,  was  run  In  School  L  and 
addressed  the  same  Issues  as  study  IV:  a)  Had  the  diagnostic  capability 
of  RPIXIE  laproved  since  study  I,  and  b)  Would  differences  arise  aaong 
conditions  vhen  the  computer  provided  the  reaedlstion? 

Procedure;  Those  students  from  study  VI  who  were  classified  as  having 
"unstable"  errors  or  who  only  had  one  pretest  score;  21  students  in  all 
were  randomly  assigned  to  three  conditions.  The  procedure  for  this 
study  was  Identical  to  that  used  in  study  IV  except  that  students  in 
this  control  group  Interacted  with  a  very  restricted  RPIXIE,  which  pro¬ 
vided  no  feedback. 

Results  and  Discussion 

Essentially,  this  study  confirmed  the  results  obtained  with  Study  IV, 
with  the  exception  that  RPIXIE  diagnosed  a  saaller  percentage  of  errors. 
This  latter  result  is  not  surprising  given  the  nature  of  this  sample 
(most  of  these  students  had  decided  not  to  study  mathematics  further.} 

VI .  Human-based  Remediation;  MBR  and  Reteaching  in  the  context  of 
stable  errors 

Given  the  apparent  instability  of  errors  suggested  by  the  analyses  of 
studies  II  and  111,  this  study  was  carried  out  to  investigate  error  sta¬ 
bility  in  depth,  and  to  compare  the  effects  of  MBR  and  Reteaching  in  the 
context  of  stable  errors. 

Subjects 


Students  from  Che  2nd-  and  3rd-  year  mathematics  classes  at  School  L 


V.  A  Third  Study  Using  RPIXIE 

An  RPIXIE  study,  analogous  to  study  IV,  was  run  In  School  L  and 
addressed  the  saae  Issues  as  study  IV:  a)  Had  the  diagnostic  capability 
of  RPIXIE  laproved  since  study  I,  and  b)  Would  differences  arise  aaong 
conditions  when  the  coaputer  provided  the  rcaedlstlon? 

Procedure:  Those  students  from  study  VI  who  were  classified  as  having 
"unstable*'  errors  or  who  only  had  one  pretest  score;  21  students  In  all 
were  randoaly  assigned  to  three  conditions.  The  procedure  for  this 
study  was  Identical  to  that  used  In  study  IV  except  that  students  In 
this  control  group  Interacted  with  a  very  restricted  RPIXIE,  which  pro¬ 
vided  no  feedback. 

Res^^ts  and  Discussion 

Essentially,  this  study  confirmed  the  results  obtained  with  Study  IV, 
with  the  exception  that  RPIXIE  diagnosed  a  smaller  percentage  of  errors. 
This  latter  result  Is  not  surprising  given  the  nature  of  this  sample 
(most  of  these  students  had  decided  not  to  study  mathematics  further.) 

I 

VI.  Human-based  Remediation:  WBR  and  Reteachlng  In  the  context  of 
stable  errors 

Given  the  apparent  instability  of  errors  suggested  by  the  analyses  of 
studies  II  and  III,  this  study  was  carried  out  to  Investigate  error  sta¬ 
bility  In  depth,  and  to  compare  the  effects  of  MBR  and  Reteachlng  In  the 
context  of  stable  errors. 

Subjects 

Students  from  the  2nd-  and  3rd-  year  mathematics  classes  at  School  L 
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served  as  the  subject  pool. 

Materials 

A  51**ltea  stability  Measure  cooslstln^  of  17  sets  of  three  algebra  tasks 
was  developed;  each  task  within  a  set  being  generated  by  the  saae  tea- 
plate  [the  template  for  the  first  set  is  aX-b;  for  set  17  it  is 
aX*b*c(dX-fe}] .  Using  these  templates,  a  pretest  and  posttest  were  con¬ 
structed,  with  the  requirement  that  the  first  item  of  each  set  on  both 
tests  be  identical.  (See  Appendix  F.) 

Procedure 

Ninety-six  students  in  the  2nd-  and  3rd-  year  classes  at  School  L  were 
pretested;  twenty-one  of  these  were  considered  not  to  need  tutoring 
because  they  had  at  least  88X  of  the  items  (AS  of  the  51)  correct.  A 
further  23  were  not  considered  because  they  would  be  absent  for  a  school 
function  during  the  week  of  tutoring.  A  further  15  students  were 
present  for  only  one  of  the  two  pretests.  Thirty-seven  students 
remained.  Of  these,  28  had  at  least  one  stable  error.  In  this  study  a 
stable  error  is  defined  as  one  that  occurs  at  least  twice  on  both 
pretests.  These  28  students  were  then  randomly  assigned  to  condition 
(KBR,  Reteachlng  or  Control). 

Students  in  the  treatment  conditions  were  seen  individually  for  a  50 
minute  period.  To  put  the  student  st  ease,  each  student  was  asked  to 
work  the  first  six  items  from  the  pretest.  After  the  first  six  items 
were  worked,  stable  errors  were  tutored.  An  error,  of  course,  was  only 
tutored  if  the  error  occurred  again  when  the  student  worked  the  item 
during  the  tutoring  session.  After  all  "stable"  errors  were  tutored, 
any  errors  made  on  the  other  51  items  were  tutored.  Students  in  the 


served  ss  the  subject  pool. 


Materlsls 

A  Sl-ltea  stsblllty  aessure  consisting  of  17  sets  of  three  slgebrs  tssks 
was  developed;  ssch  tssk  within  a  set  being  generated  by  the  saae  tea- 
plate  (the  template  for  the  first  set  is  aX~b;  for  set  17  it  is 
aX-b*c(dX4-e)] .  Using  these  templates,  a  pretest  and  posttest  were  con¬ 
structed,  with  the  requlreaent  that  the  first  item  of  each  set  on  both 
tests  be  identical.  (See  Appendix  F.) 

Procedure 

Ninety-six  students  in  the  2nd-  and  3rd-  year  classes  at  School  L  were 
pretested;  twenty-one  of  these  were  considered  not  to  need  tutoring 
because  they  had  at  least  88Z  of  the  items  (45  of  the  51)  correct.  A 
further  23  were  not  considered  because  they  would  be  absent  for  a  school 
function  during  the  week  of  tutoring.  A  further  15  students  were 
present  for  only  one  of  the  two  pretests.  Thirty-seven  students 
remained.  Of  these,  26  had  ar  least  one  stable  error.  In  this  study  a 
stable  error  is  defined  as  one  that  cccurs  at  least  twice  on  both 
pretests.  These  28  students  were  then  randomly  assigned  to  condition 
(MBR,  Reteschlng  or  Control). 

Students  in  the  treatment  conditions  were  seen  individually  for  a  50 
alnute  period.  To  put  the  student  at  ease,  each  student  was  asked  to 
work  the  first  six  items  from  the  pretest.  After  the  first  six  items 
were  worked,  stable  errors  were  tutored.  An  error,  of  course,  was  only 
tutored  if  the  error  occurred  again  when  the  student  worked  the  item 
during  the  tutoring  session.  After  all  ’’stable"  errors  were  tutored, 
any  errors  made  on  the  other  51  items  were  tutored.  Students  in  the 


Control  group  limply  took  Che  pretests  end  posttest.  The  week  sfter 
tutoring,  •  posttest  was  given  to  Che  classes  Involved.  A  delayed  post- 
test  was  also  given  a  month  after  the  Initial  posttest. 

Results 

Of  the  28  students,  3  were  absent  from  school  -  one  from  each  condition 
-  when  the  first  posttest  was  given,  resulting  In  the  following  cell 
slses:  KBR  9,  Reteachlng  8,  Control  8.  In  addition,  one  student  In  KBR 
was  subsequently  found  during  a  reliability  check  to  have  no  stable 
errors.  His  data  are  Included  In  the  analyses  because  they  are  other¬ 
wise  sound.  The  same  three  students  and  one  other  from  the  KBR  condi¬ 
tion,  were  absent  from  school  on  the  day  of  the  delayed  posttest,  leav¬ 
ing  eight  students  per  condition. 


Table  5  -  Mean  Pretest  and  Poattest  Scores*  by  Condition 


Condition  Pretest  1  Pretest  2  Posttest  1  Delayed  Posttest 


Reteach 


Control 


27.50 

(12.751 


28.00 

(14.49) 


23.44 

(12.35) 


29.70 

(14.04) 


33.56 

(11.06) 


25.44 

(13.44) 


41.22 

(9.09) 


26.00 

(14.36) 


45.38 

(5.55) 


28.38 

(15.15) 


*Maxlttum  *51 


Table  5  gives  the  aeans  and  standard  deviation  by  condition  for  the  four 
testings.  An  ANOVA  showed  no  significant  differences  anong  the  condi¬ 
tions  (N>28)  on  either  pretestl:  F  (2,25)>.33,  £  >.72,  or  pretest2: 
(2,25)«.86,  2  >.42,  established  that  randomization  had  been  successful. 


Significant  omnibus  F  ratios  were  found  for  both  the  posttest  (N*25)  and 
delayed  posttests  (N-24):  F  (2,22)-6.33,  £  <.008,  and  F  (2,21)-5.78,  £ 
<.02,  respectively.  Post  hoc  analyses  using  the  Scheffe  test  showed  no 
differences  (  £  >.05)  between  KBR  and  Reteschlng,  but  both  being  better 
(  £  <.05)  than  the  control  condition  on  both  posttests.  Although,  ANOVA 
Is  generally  regarded  to  be  robust  with  respect  to  violations  of  Its 
assumptions,  the  reader  should  note  the  small  cell  sizes  and  the  fact 
that  heterogeneity  of  variance  (  £  <.05)  was  demonstrated  for  both  post- 
tests  using  Bartlett's  test. 


A  paired  t-test  comparing  KBR  and  Reteachlng  scores  on  pretestl  with 
matched  scores  on  the  posttest  showed  s  significant  gain  from  a  mean  of 
27.35  to  one  of  41.41,  t(16)>-3.70,  p<.001.  However,  the  reader  should 
note  that  the  correlated  variances  (192.38  and  50.13,  respectively)  were 
found  to  be  significantly  different:  t(l5)-3.28,  p<.01.  A  similar 
analysis  for  Che  control  condition  showed  no  significant  differences 


between  either  Che  aeane  on  pretestl  end  Che  poettest:  c(7}>-1.77, 
p<.12,  or  Che  correleced  verlences  (124.77  end  206.21,  respectively): 
t(6)— 0.5919,  p>.l. 


Anelvses  of  Sceblllt) 


The  cotel  set  of  errors  encountered  In  this  cxperlaent,  were  clesslfled 
Into  46  different  types.  Only  19  of  these  types  appeared  during  tutor¬ 
ing;  the  ocher  27  types  occurred  Infrequently  on  the  pretests  (l.e., 
were  not  stable)  or  only  on  Che  posttest. 


Table  6:  Number 

of  times 

Che  19  error 

types  occurred 

Condition 

Pretest 

1  Pretest  2 

Postteat 

MBR 

173 

167 

93 

Reteachlng 

235 

181 

79 

Control 

256 

211 

208 

An  inapectlon  of  Table  6  ahows  that  for  all  three  condltlona  there  was  a 
decrease  In  the  number  of  errors  from  pretestl  to  pretest2.  The  percen¬ 
tage  decrease  by  condition  from  pretestl  to  prctcBt2  and  from  pretestl 
to  posttest  are  shown  In  table  7. 


Table  7:  Percentage  Decrease  In  the  19  Stable  Errors  by  Condition. 

Prel  tc  Pre2  Prel  to  Post 

MBR  3.5  46.2 

Reteachlng  23.0  66.4 

Control  17.6  18.75 

[Note;  There  was  an  overall  average  decrease  of  1S.8Z  from  pretestl  to 
pretest2.] 

Table  7  shows  chat  Che  percentage  decrease  for  Che  control  condition 
remains  basically  unchanged,  whereas  for  the  treatment  conditions  the 
percentage  decreases  are  much  more  dramatic,  mirroring  Table  6.  (This 
result  Is  completely  consistent  with  the  result  reported  by  Sleeman 
(1983)  for  e  study  Involving  2  groups:  essentially  MBR  and  Control.) 


Table  Analyalt  of  the  19  ateble  errors. 

Nuaber  Prevalence  Freq.  of 

Students  of  stable  errors  on 

who  had  a  error  Pretest  1 

particular  (as  Z  of  (as  Z) 

(stable)  student 

Error  Type  error  population 


Freq.  of  Freq.  of 
errors  on  errors  on 

Pretest  1  Pretest  2 

(as  Z)  (as  Z) 


1  Bracket 

2  Precedence 

3  CoBputatlonal 

4  Change  side  not  sign  of 
x-term 

5  Subtract  a  coeff 

6  Add  X  and  constant 

7  Add  a  negative  sign 

8  Uses  a  nuaber  twice 

9  Drop  an  X 

10  Inverted  division 

11  Minus  before  the 
wrong  number 

12  Subtract  a  multiplier 

13  Subtract  a  multiplied 
x-term 

14  ax  -  b  ->  X  -  a  -  b 

15  Drop  a  negative  sign 

16  Multiply  across  by 
a  coeff 

17  Ends  task  with  ax  ■  bx 

18  ax  •  b  ■>  X  ■  -  (a  +  b 

19  a  *  bx  +  cx  •  d  -> 
a  *  (-d)  •  “bx  “cx 


Table  8  shows  that  the  percentage  of  students  who  made  stable  errors 
varies  by  type  of  error.  Some  stable  errors  (e.g.,  precedence  errors) 
are  more  prevalent  than  others  (e.g.,  'Inverted  division").  The  rela¬ 
tionship  between  the  relative  prevalence  of  a  stable  error  and  the  rela¬ 
tive  frequency  with  which  It  occurs  In  the  population  of  errors  Is  not 
one-to-one.  For  example,  precedence  errors  occur  in  S2Z  of  the  sample 
of  students,  yet  account  for  only  lOZ  of  the  total  pretestl  errors; 
Inverted  division  errors  occur  In  only  8Z  of  the  student  sample,  but 
account  for  13Z  of  the  pretestl  errors.  (Note  further  that  these  fre¬ 
quency  figures  are  NOT  the  same  for  both  pretests.) 
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Noise 

In  addition  to  these  stable  errors,  there  were  errors  on  the  pretests 
that  were  either  unstable  (by  the  definition  (Iven  above),  or  undiag¬ 
nosed.  These  latter  errors  together  with  those  labelled  "coaputatlonal" 
in  Table  8  will  be  described  collectively  as  ‘'noise".  "Noise*  accounted 
for  16Z  of  the  pretestl  errors,  and  for  14Z  of  the  pretest!  errors.  By 
way  of  comparison,  Sleeman  (1982)  reported  29. 6Z  noise  for  a  student 
population  when  It  interacted  with  a  predecessor  of  RPIXIE.  (Suggesting 
that  students'  responses  In  this  domain  are  noisier  when  Interacting 
with  programs  than  when  interacting  with  human  tutors. 

If  one  extended  the  definition  of  noise  to  include  hose  stable  errors 
made  by  only  a  small  percentage  of  the  sample.  The  percentages  of  total 
errors  described  as  noise  would  rise  to  22Z  on  pretestl  (18Z  on 
pretest!)  if  one  Included  stable  errors  made  by  4Z  or  less  of  the  stu¬ 
dent  sample.  The  percentages  would  rise  to  43Z  on  pretestl  (38Z  on 
pretest!)  If  one  Included  stable  errors  made  by  8Z  or  less  of  the  stu¬ 
dent  sample. 

How  many  stable  errors  did  the  students  have? 

Table  9  shows  the  number  of  students  who  had  at  least  one  stable  error, 
at  least  two  stable  errors,  and  so  on  up  to  eight  stable  errors. 
Clearly,  most  students  have  at  least  two  stable  errors,  but  the  percen¬ 
tage  drops  off  sharply. 


Table  9:  Number  of  etudente  with  N  Stable  errors 
Number  of 

stable  1  2  3  4  5  6  7  8 

errors 


Number  of 

students  24  22  9  7  4  2  1  0 

Remediating  Stable  Errors  by  Conditions 

If  one  Hats  by  condition  the  number  of  stable  errors  which  occurred  for 
all  3  conditions  during  the  pretests,  and  compares  this  with  the  number 
of  errors  that  appeared  on  the  posttest,  one  can  express  the  remedial 
effectiveness  of  a  condition  as  the  percentage  reduction  of  errors,  see 
Table  1C.  (Note,  not  all  stable  errors  occurred  in  each  of  tht  condi¬ 
tions,  and  so  these  groups  are  only  partially  "matched*') . 

Table  10:  Remediating  Stable  Errors  which  occur  on  all  2  conditions 

Stable  Errors 


Condition 

Number  on  Pretests 

Still  on  posttest 

X  reduction 

KBR 

10 

5 

50 

Reteachlng 

12 

7 

42 

Control 

12 

12 

0 

Note:  Of  the  six  categories  of  stable  errors  In  this  table,  only  bracket 
(distributive  law)  errors  and  precedence  errors  appeared  on  the  posttest 
for  both  conditions.  We  label  these  errors,  "resistant". 


An  Inapectlon  of  Table  10  reflects  the  overall  findings  of  this  study: 
both  treatment  conditions  whilst  outperforming  the  control  condition  are 
themselves  very  comparable. 


Discussion 


Effects  of  different  forms  of  Reaedlstlon 

Once  again,  study  VI  has  shown  that  MBR  and  Reteachlng  are  both  better 
than  no  treatment,  but  we  have  been  unable  to  distinguish  between  the 
two.  So  with  this  subject  domain  when  taught  procedurally  with  this 
type  of  student  population,  we  concluded  that  even  when  one  Is  remediat¬ 
ing  stable  errors,  MBR  and  Reteachlng  appear  to  be  equally  effective. 
This  Is  still  a  surprising  result  to  us.  Below,  we  speculate  further 
about  this  result: 

1.  Maybe  the  MBR  and  Reteachlng  treatments  used  In  this  study  were 
still  too  similar.  Perhaps,  the  scripts  followed  were  not  suffi¬ 
ciently  different  (In  particular,  we  noted  above  that  the  MBR  -f 
Cognitive  Dissonance  treatment  should  be  rerun,  as  the  students  In 
study  Ill  did  not  have  an  appropriate  background).  Another  possi¬ 
ble  reason  for  lack  of  effect,  is  the  duration  of  the  exposure. 
Fifty  minutes  of  tutorial  seemed  a  substantial  period;  however. 
Swan  (1983)  reports  effects  after  eight  one-hour  long  lessons. 

2.  Even  though  the  MBR  and  Reteachlng  groups  were  highly  comparable  on 
procedural  tasks,  had  they  been  tested  for  conceptual  understanding 
of  algebra  we  might  have  found  that  the  MBR  atudents  would  have 
outperformed  the  Reteachlng  students. 

3.  As  noted  In  the  Introduction,  MBR  essentially  assumes  that  the  stu¬ 
dent  has  a  (stable)  mental  model,  to  which  remedial  comments  are 
related.  So  an  additional  hypothesis  we  created.  Is  that  students 
taught  algebra  procedurally  have  a  (weak)  mental  model  of  the 


domain.  Further,  we  have  hypothesized  that  students  taught  algebra 
conceptually  should  have  a  much  stronger  mental  model,  and  so  one 


should  observe  with  such  students  significant  differences  between 
the  KBR  and  Reteachlng  treatments.  Unfortunately,  we  were  unable 
to  find  a  hlgh-achool  In  Aberdeen  where  algebra  was  taught 


(largely)  conceptually,  so  this  hypothesis  remains  untested.* 


A  further  speculation  is  that  the  MBR  and  Reteachlng  treatments  are 


in  fact  very  similar,  because  each  student  in  the  Reteachlng  treat¬ 
ment,  essentially  notes  for  himself  the  difference  between  his 
answer  and  that  provided  by  the  correct  procedure,  and  so  generates 


his  own  MBR.  Although  this  is  of  little  importance  for  instruc¬ 
tion,  we  plan  to  run  an  experiment  to  probe  this  issue.  (If  this 
is  the  mechanism,  one  would  expect  to  find  differences  between  the 
mathematically  able  and  less-able  students.) 


*  Kelly  and  Sleeman  (1986)  found  that  most  teachers 
teach  algebra  with  a  rule-based  emphasis.  Because  the 
nature  of  the  student's  initial  Instruction  became  a 
concern  for  the  PIXIE  studies,  we  looked  for  a  school 
that  taught  algebra  conceptually.  Unfortunately,  no 
such  school  was  available  in  Aberdeen.  However,  we  did 
Interview  atudents  to  determine  their  understanding  of 
algebra.  The  interview  questionalre  was  a  revised  ver¬ 
sion  of  a  questlonaire  developed  by  Sleeman,  Stelnburg 
A  Ktorza  (1985).  We  found  that  even  students'  concep¬ 
tual  understanding  of  algebra  ia  inconsistent  within  a 
one-hour  interview  period.  Indeed  this  experiment  has 
led  us  to  the  not-too-surprising  position  that  concep¬ 
tual  understanding  is  multifaceted  (as  is  procedural 
competence)  and  not  monolithic  as  is  often  assumed. 


we? 
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Stability  of  Errors 


The  asBunptlon  that  student  errors  are  conpletely  atable  has  been 
clearly  questioned  by  several  studies.  Including  VsnLehn  (1981);  Brlcken 
(1987)  as  well  as  Slccaan  (1983)**  The  following  Issues  on  error  sta¬ 
bility  are  raised  by  the  results  of  study  VI: 

1.  Attentions!  nature  of  some  errors.  Forty-six  types  of  errors  were 
encountered  during  the  study,  but  only  19  of  these  were  '‘stable'* 
during  the  tutorial  sessions.  This  suggests  that  tutorial  contact 
la  effective  In  getting  rid  of  soae  types  of  careless  errors,  such 
as  dropping/adding  signs.  (The  explanation  for  this  appears  to  be 
the  aotlvatlonal  effect  of  having  a  tutor  work  with  a  student.) 

2.  Prevalence  and  Frequency.  We  have  Introduced,  and  wish  to  stress, 
the  two  aeasures  we  have  used  to  describe  this  phenoaenon.  Pre¬ 
valence  Indicates  the  nuaber  of  students  In  the  population  who  have 
a  particular  error.  Frequency  Indicates  the  proportion  of  the 
total  nuaber  of  errors  that  are  explained  by  the  specific  error. 
Both  aeasures  are  needed  to  discuss  the  phenoaena  of  errors. 

3.  Use  of  this  data  for  reaedlatlon:  A  class  teacher  will  be  essen- 


**VanLehn  (1981)  Investigated  short-tera  and  long-term 
stability  of  subtraction  errors,  and  found  chat  only 
12Z  of  Che  students  who  had  errors  on  the  first  test 
had  the  same  errors  on  the  second  (short  tens)  test. 
He  also  found  that  the  long  Cera  stability  data  are 
very  slallar  to  the  short  tera  stability  data.  VanLehn 
concluded  Chat  errors  In  general  are  not  atable. 
Further,  Brlcken  (1987)  Inveatlgated  the  stability  of 
algebra  errors,  and  found  that  SOX  (11  of  22  students) 
coaaltted  at  least  one  error  on  the  pretest  which  reoc- 
currcd  during  their  Interview  held  two  weeks  later. 
Although  the  studies  aentloned  above  all  conflra  that 
errors  generally  are  not  stable.  It  is  laportant  to 
note  that  each  Investigation  aeasures  stability  using 
different  criteria.  However,  lack  of  stability  seems 
to  be  an  Issue  regardless  of  how  It  Is  aeasured. 


tially  concerned  to  know  the  aoet  coononly  occurlng  (l«e.,  the  nost 
prevalent )  errora  In  the  claaa.  Whereas,  a  tutor  will  wish  to  have 
access  to  the  actual  arror-profile  for  each  student. 


4.  Taxonomy  of  errors:  This  study  has  confirmed  and  slightly  extended 
the  error  taxonomy  suggested  by  Sleeman  (1983).  In  our  view,  it  is 
feasible  to  talk  about  the  following  types  of  errors: 

-  Stable  errors  (both  remediable  and  ‘‘resistant”) 

-  Attentlonal  errors  (largely  minor  errors  such  as  adding/dropping 
signs) . 

-  Classes  of  mal~rules,  used  by  the  same  student  different  occa¬ 
sions  with  Che  same  type  of  task. 

>  Mental  slips,  typlng/transcrlptlon  errors. 

It  is  important  for  a  tutor  to  categorise  correctly  the  error  as, 
for  example,  it  may  be  counterproductive  to  tutor  a  student  on  an 
error  which  is  a  result  of  a  slip;  whereas  it  maybe  important  to 
address  a  stable  error.  (These  judgements  are  both  subtle  and  com¬ 
plex.) 


C"* 
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Overall  conclusions  of  the  series  of  studies 


For  Introductory  algebra  when  taught  procedurally  with  this  type  of 
student.  It  appears  that  fieteachlng  is  as  affective  as  KBR.  From 
which  it  follows  that  CAI  would  be  as  affective  as  an  ITS  It  is 
vital  that  we  Investigate  the  range  of  subjects.  Instructional 
approachea  and  student  age-ranges  for  which  this  result  holds. 
(This  InfonBation  will  be  important  for  both  educators  and  ITSs 
workers. ) 

Despite  Che  conclusions  on  the  last  paragraph,  one  should  not  con¬ 
clude  that  reteachlng  by  a  classroom  teacher  will  be  as  effective 
as  reteachlng  in  a  one-to-one  tutoring  situation  or  by  a  computer 
system.  Immediacy  of  feedback  which  is  lacking  from  the  normal 
classroom  situation  may  well  be  a  critical  factor,  Lewis  fc  Anderson 
(1985). 


A 


It  is  critical  that  ITSs  receive  extensive  field-testing. 

The  subflcld  of  ITS  should  not  conclude  thst  the  task  of  building 
an  ITS  is  impossible,  but  it  should  conclude  that  the  task  is 
harder  than  we  had  initally  thought*,  c.f.,  Wlnograd  &  Flores 
(1966).  It  is  possible  that  the  more  global  analysis  (Moore  & 
Slecman,  1987)  which  takes  account  of  the  student's  performance  at 


several  levels  might  be  needed  before  proper  remediation  can  be 
undertaken.  But  before  such  a  system  is  built  it  is  suggested  that 


experiments  are  run  to  see  whether  human  tutors  are  more  affective 


when  they  base  their  remediation  on  such  a  global  approach. 


*  Cronbach  and  Snow  (1977)  warned  that  producing  truly 
Individualised  instruction  was  a  demanding  task,  but 
the  ITS  field  chose  to  infer  that  their  conclusions 
were  unduly  pessimistic  given  the  complexity  of  analys¬ 
ing  such  a  diverse  set  of  experimental  studies. 
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For  a  aore  cxteoslve  discusalon  of  the  conclusions  of  this  experiment 
including  e  discussion  of  possible  additional  Issues  to  be  investigated, 
see  Sleeman,  Kelly,  Martlnak,  Ward  and  Moore  (19B7). 
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Pretest  Items 


Appendix  A 


Posttest  Items 


7x  -  14 

1. 

3x  -  9 

4x  -  -12 

2. 

4x  -  -8 

5x  -  7 

3. 

3x  -  5 

8x  -18 

4. 

6x  -  9 

3x  -  8  +  4 

5. 

5x  -  8  +  2 

3x  -  4  *  4 

6. 

3x  -  4  *  3 

3x  +  4x  -  14 

7. 

2x  +  3x  -  10 

3x  +  5  ■  26 

8. 

2x  +  3  +  9 

7  +  4x  -  19 

9. 

2  +  4x  -  14 

5x  -  4x  +  8 

10. 

4x  +  3x  +  6 

3x  -  3(2  +  3) 

11. 

2x  -  3(3  +  1) 

12x  -  2(3x  +  3) 

12. 

24x  -  3(2x  +  3) 

7x  +  2  +  4  *  8 

13. 

5x  -  2  *  3  *  8 

17x  -  19x  +  25 

14. 

16x  -  19x  +  20 

3  +  2x  +  4x  -  21 

15. 

2  +  2x  +  3x  -  17 

4  +  3x  +  4x  -  25 

16. 

3  +  3x  +  4x  -24 

35x  -  2  +  3(4x  +  5) 

17. 

37x  -  2  +  3(4x  + 

Control  group  almply  took  Che  pretests  and  posttest.  The  week  after 
tutoring,  a  postteat  was  given  to  the  classes  Involved.  A  delayed  post¬ 
test  was  also  given  a  wonth  after  the  Initial  posttest. 

Results 

Of  the  28  students,  3  were  absent  from  school  -  one  from  each  condition 
-  when  the  first  posttest  was  given,  resulting  In  the  following  cell 
sizes:  MBR  9,  Reteachlng  8,  Control  8.  In  addition,  one  student  in  MBR 
was  subsequently  found  during  a  reliability  check  to  have  no  stable 
errors.  His  data  are  Included  In  the  analyses  because  they  are  other¬ 
wise  sound.  The  same  three  students  and  one  other  from  the  KBR  condi¬ 
tion,  were  absent  from  school  on  the  day  of  the  delayed  posttest,  leav¬ 


ing  eight  students  per  condition. 


Table  5  - 

Mean  Pretest 

and  Posttest 

Scores* 

by  Condition 

Condition 

Pretest  1 

Pretest  2 

Posttest  1  Delayed  Posttest 

KBR 

27.50 

29.70 

41.22 

45.36 

(12.79) 

(14.04) 

(9.09) 

(5.55) 

Reteach 

28.00 

33.56 

41.62 

37.12 

(14.49) 

(11.06) 

(4.47) 

(8.69) 

Control 

23.44 

25.44 

26.00 

28.38 

(12.35) 

(13.44) 

(14.36) 

(15.15) 

^Kaximua  ■ 

-  51 

Table  5  gives  the  Besaa  and  standard  deviation  by  condition  for  the  four 
'estings.  An  ANOVA  showed  no  significant  differences  aaong  Che  condi¬ 
tions  (N*28)  on  either  pretestl:  (2,2S)>.33,  £  >.72,  or  pretesc2: 

(2,25)b.88,  _£  >.42,  established  that  randomization  had  been  successful. 

Significant  omnibus  F  ratios  were  found  for  both  Che  posttest  (N-25)  and 
delayed  posttests  (N-24):  F  (2,22)-6.33,  £  <.008,  and  F  (2,21)-5.78,  £ 
<.02,  respectively.  Post  hoc  analyses  using  the  Scheffe  test  showed  no 
differences  (  £  >.05)  between  KBR  and  Reteachlng,  but  both  being  better 
(  £  <.05}  than  Che  control  condition  on  both  posttests.  Although,  ANOVA 
Is  generally  regarded  to  be  robust  with  respect  to  violations  of  its 
assumptions,  the  reader  should  note  the  small  cell  sizes  and  the  fact 
that  heterogeneity  of  variance  (  £  <.05)  was  demonstrated  for  both  post- 
tests  using  Bartlett's  test. 

A  paired  t-test  comparing  KBR  and  Reteachlng  scores  on  pretestl  with 
matched  scores  on  the  posttest  showed  a  significant  gain  from  a  mean  of 
27.35  to  one  of  41.41,  t(16)*-3.70,  p<.001.  However,  the  reader  should 
note  that  the  correlated  variances  (192.38  and  50.13,  respectively)  were 
found  to  be  significantly  different:  t(15)-3.26,  p<.01.  A  similar 

analysis  for  the  control  condition  showed  no  significant  differences 


between  either  the  Beans  on  pretestl  and  the  poattest:  t(7}— 1.77, 
p<.12,  or  the  correlated  variances  (124.77  and  206.21,  respectively): 
t(6)— 0.5919,  p>.l. 


Analyses  of  Stability 

The  total  set  of  errors  encountered  in  this  experiment,  were  classified 
Into  46  different  types.  Only  19  of  these  types  appeared  during  tutor¬ 
ing;  the  other  27  types  occurred  Infrequently  on  Che  pretests  (l.e., 
were  not  stable)  or  only  on  the  posttest. 


Table  6:  Number 

of  times 

the  19  error 

types  occurred 

Condition 

Pretest 

1  Pretest  2 

Posttest 

MBR 

173 

167 

93 

Beteachlng 

235 

181 

79 

Control 

256 

211 

208 

An  inspection  of  Table  6  shows  that  for  all  three  conditions  there  was  a 
decrease  in  the  number  of  errors  from  pretestl  to  pretest2.  The  percen¬ 
tage  decrease  by  condition  from  pretestl  to  pretest2  and  from  pretestl 
to  posttest  are  shown  in  table  7. 


Table  7 :  Percentage  Decrease  in  the  19  Stable  Errors  by  Condition. 

Prel  to  Pre2  Prel  to  Post 

MBR  3.5  46.2 

Beteachlng  23.0  66.4 

Control  17.6  18.75 

[Note:  There  was  an  overall  average  decrease  of  15. 8X  from  pretestl  to 
pretest2. ] 

Table  7  shows  that  the  percentage  decrease  for  the  control  condition 
remains  basically  unchanged,  whereas  for  the  treatment  conditions  the 
percentage  decreases  are  much  more  dramatic,  mirroring  Table  6.  (This 
result  is  completely  consistent  with  the  result  reported  by  Sleeman 
(1983)  for  a  study  involving  2  groups:  essentially  MBR  and  Control.) 
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Table  8:  Analysis  of  the  19  stable  errors. 

Nuaber 

Prevalence 

Freq.  of 

Freq. 

Students 

of  stable 

errors  on 

errors 

who  had  a 

error 

Pretest  1 

Pretest 

particular 

(as  Z  of 

(as  Z) 

(as  Z) 

(stable) 

student 

Error  Type 

error 

population 

1 

Bracket 

14 

56 

10 

12 

2 

Precedence 

13 

52 

10 

8 

3 

Coaputatlonal 

7 

28 

9 

7 

4 

Change  side  not  sign  of 

z-tera 

5 

20 

7 

7 

5 

Subtract  a  coeff 

4 

16 

16 

18 

c 

Add  X  and  constant 

3 

12 

4 

7 

7 

Add  a  negative  sign 

3 

12 

2 

2 

8 

Uses  a  nuaber  twice 

3 

12 

3 

5 

9 

Drop  an  Z 

2 

8 

3 

3 

10 

Inverted  division 

2 

8 

13 

8 

11 

Minus  before  the 

wrong  nuaber 

2 

8 

3 

4 

12 

Subtract  a  aultlpller 

2 

8 

1 

1 

13 

Subtract  a  aultlplled 

x-term 

2 

8 

2 

3 

14 

ax«b«>x*a-b 

2 

8 

1 

1 

15 

Drop  a  negative  sign 

1 

4 

3 

2 

16 

Multiply  across  by 

a  coeff 

1 

4 

2 

1 

17 

Ends  task  with  ax  ■  bx 

1 

4 

1 

1 

18 

ax«b»>x"-(a+b 

1  * 

4 

0 

0 

19 

a  *  bx  +  cx  ■  d  •> 

' 

a  *  (-d)  •  -bx  -cx 

1 

4 

0 

0 

Table  6  shows  that  the  percentage  of  students  who  wade  stable  errors 
varies  by  type  of  error.  Soae  stable  errors  (e.g.,  precedence  errors) 
are  aore  prevalent  than  others  (e.g.,  "inverted  division").  The  rela¬ 
tionship  between  the  relative  prevalence  of  a  stable  error  and  the  rela¬ 
tive  frequency  with  which  It  occurs  In  the  population  of  errors  Is  not 
one-to-one.  For  cxaaple,  precedence  errors  occur  in  52X  of  the  saaple 
of  students,  yet  account  for  only  lOZ  of  the  total  pretest 1  errors; 
inverted  division  errors  occur  in  only  8Z  of  the  student  saaple,  but 
account  for  13Z  of  the  pretestl  errors.  (Note  further  that  these  fre¬ 
quency  figures  are  NOT  the  saae  for  both  pretests.) 


Noise 


In  addition  to  these  stable  errors,  there  were  errors  on  the  pretests 
that  were  cither  unstable  (by  the  definition  given  above),  or  undiag¬ 
nosed.  These  latter  errors  together  with  those  labelled  ‘'coaputatlonal*' 
In  Table  6  will  be  described  collectively  as  'noise'.  'Noise'  accounted 
for  16Z  of  the  pretestl  errors,  and  for  14Z  of  the  pretest!  errors.  By 
way  of  coaparlson,  Sleeman  (1982)  reported  29. 6Z  noise  for  a  student 
population  when  It  Interacted  with  a  predecessor  of  KFIXIE.  (Suggesting 
that  students'  responses  in  this  donaln  are  noisier  when  interacting 
with  programs  than  when  Interacting  with  human  tutors. 

If  one  extended  the  definition  of  noise  to  Include  those  stable  errors 
made  by  only  a  small  percentage  of  the  sample.  The  percentages  of  total 
errors  described  as  noise  would  rise  to  22Z  on  pretestl  (18Z  on 
pretest2)  if  one  Included  stable  errors  made  by  4Z  or  less  of  the  stu¬ 
dent  sample.  The  percentages  would  rise  to  4SZ  on  pretestl  (38Z  on 
pretestl)  if  one  Included  stable  errors  made  by  8Z  or  less  of  the  stu¬ 
dent  sample. 

How  many  stable  errors  did  the  students  have? 

Table  9  shows  the  number  of  students  who  had  at  least  one  stable  error, 
at  least  two  stable  errors,  and  ao  on  up  to  eight  stable  errors. 
Clearly,  most  atudents  have  at  least  two  stable  errors,  but  the  percen¬ 
tage  drops  off  sharply. 


>.v 
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Table  9:  Wuaber  of  atudenta  with  1)  Stable  errors 
Number  of 

stable  12345678 

errors 


Number  of 

students  24  22  9  7  4  2  1  0 

Remediating  Stable  Errors  by  Conditions 

If  one  llsta  by  condition  the  number  of  stable  errors  %ihlch  occurred  for 
all  3  conditions  during  Che  pretests,  and  compares  this  with  Che  number 
of  errors  that  appeared  on  the  poattest,  one  can  express  the  remedial 
effectiveness  of  a  condition  as  the  percentage  reduction  of  errors,  see 
Table  10.  (Note,  not  all  stable  errors  occurred  In  each  of  the  condi¬ 
tions,  and  so  these  groups  are  only  partially  "matched''). 


Table  10: 

Remediating  Stable  Errors 

which 

occur  on  all 

3  conditions 

Stable 

Errors 

Condition 

Number  on  Pretests 

Still 

on  posttest 

X  reduction 

KBR 

10 

5 

50 

Reteachlng 

12 

7 

42 

Control 

12 

12 

0 

Note:  Of  the  six  categories  of  ateble  errors  in  this  table,  only  bracket 
(distributive  law)  errors  and  precedence  errors  appeared  on  the  posttest 
for  both  conditions.  We  label  these  errors,  "resistant". 


An  inspection  of  Table  10  reflects  the  overall  findings  of  this  study: 
both  treatment  conditions  whilst  outperforming  Che  control  condition  are 
themselves  very  comparable. 


PlBCUBSlon 


Effects  of  different  forms  of  Remedlstlon 

Once  again,  study  VI  has  shovn  chat  MBR  and  Reteachlng  are  both  better 
than  no  treatment,  but  we  have  been  unable  to  distinguish  between  the 
two.  So  with  this  subject  domain  «fhen  taught  procedurally  with  this 
type  of  student  population,  we  concluded  that  even  when  one  Is  remediat¬ 
ing  stable  errors,  KBR  and  Reteachlng  appear  to  be  equally  effective. 
This  Is  still  a  surprising  result  to  us.  Below,  we  speculate  further 
about  this  result: 

1.  Maybe  the  KBR  and  Reteachlng  treatments  used  In  this  study  were 
still  too  similar.  Perhaps,  the  scripts  followed  were  not  suffi¬ 
ciently  different  (In  particular,  we  noted  above  that  the  MBR  -f 
Cognitive  Dissonance  treatment  should  be  rerun,  as  the  students  In 
study  III  did  not  have  an  appropriate  background).  Another  possi¬ 
ble  reason  for  lack  of  effect.  Is  Che  duration  of  the  exposure. 
Fifty  minutes  of  tutorial  seemed  a  substantial  period;  however. 
Swan  (1983)  reports  effects  after  eight  one-hour  long  lessons. 

2.  Even  though  the  MBR  and  Reteachlng  groups  were  highly  comparable  on 
procedural  tasks,  had  they  been  tested  for  conceptual  understanding 
of  algebra  we  might  have  found  that  the  MBR  students  would  have 
outperformed  the  Reteachlng  students. 

3.  As  noted  In  Che  Introduction,  MBR  essentially  assumes  that  the  stu¬ 
dent  has  a  (stable)  mental  model,  to  which  remedial  comments  are 
related.  So  an  additional  hypothesis  we  created.  Is  that  students 
taught  algebra  procedurally  have  a  (weak)  mental  model  of  the 


r. 


l>“v' 
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douln.  Further,  ve  have  hypotheeleed  that  etudente  caught  algebra 
copceptually  should  have  a  such  stronger  aental  aodel,  and  so  one 
should  observe  with  such  students  significant  differences  between 
the  MBR  and  Reteachlng  treataents.  Unfortunately,  we  were  unable 
to  find  a  hlgh-school  In  Aberdeen  where  algebra  was  taught 
(largely)  conceptually,  so  this  hypothesis  remains  untested.* 

4.  A  further  speculation  Is  that  the  MBR  and  Reteachlng  treataents  are 
in  fact  very  similar,  because  each  student  in  the  Reteachlng  treat¬ 
ment,  easentlally  notes  for  himself  the  difference  between  his 
answer  and  that  provided  by  the  correct  procedure,  and  so  generates 
his  own  MBR.  Although  this  is  of  little  importance  for  Instr  iC- 
tlon,  we  plan  to  run  an  experiment  to  probe  this  Issue.  (If  this 
Is  the  mechanism,  one  would  expect  to  find  differences  between  the 
mathematically  able  and  less-able  students.) 


*  Kelly  and  Slceaan  (1986)  found  that  most  teachers 
teach  algebra  with  a  rule-based  emphasis.  Because  the 
nature  of  the  student's  Initial  Instruction  became  a 
concern  for  the  PIXIE  studies,  we  looked  for  a  school 
that  taught  algebra  conceptually.  Unfortunately,  no 
such  school  wss  available  In  Aberdeen.  However,  we  did 
Interview  students  to  determine  their  understanding  of 
algebra.  The  Interview  questlonaire  was  a  revised  ver¬ 
sion  of  a  questlonaire  developed  by  Sleeman,  Steinburg 
A  Ktorsa  (1985).  Ve  found  that  even  students'  concep¬ 
tual  understanding  of  algebra  is  inconsistent  within  a 
one-hour  interview  period.  Indeed  this  experiment  has 
led  us  to  the  not-too-surprising  position  that  concep¬ 
tual  understanding  is  multifaceted  (as  Is  procedural 
competence)  and  not  monolithic  as  Is  often  assumed. 
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Stability  of  Errors 

The  aesuaptlon  that  atudent  errors  are  completely  atable  has  been 
clearly  questioned  by  several  studies,  including  VanLehn  (19B1);  Brlcken 
(1987)  as  well  as  Sleeman  (1983)**  The  following  Issues  on  error  sta¬ 
bility  are  raised  by  the  results  of  study  VI: 

1.  Attentions!  nature  of  some  errors.  Forty-six  types  of  errors  were 
encountered  during  the  study,  but  only  19  of  these  were  ’stable*' 
during  Che  tutorial  sessions.  This  suggests  that  tutorial  contact 
is  effective  in  getting  rid  of  some  types  of  careless  errors,  such 
as  dropping/adding  signs.  (The  explanation  for  this  appears  to  be 
the  motivational  effect  of  having  a  tutor  work  with  a  student.) 

2.  Prevalence  and  Frequency.  We  have  introduced,  and  wish  to  stress, 
the  two  measures  we  have  used  to  describe  this  phenomenon.  Pre¬ 
valence  indicates  the  number  of  students  in  Che  population  who  have 
a  particular  error.  Frequency  indicates  the  proportion  of  the 
total  number  of  errors  that  are  explained  by  the  specific  error. 
Both  measures  are  needed  to  discuss  the  phenomena  of  errors. 

3.  Use  of  this  data  for  remediation:  A  class  teacher  will  be  essen- 


**VanLehn  (1981)  investigated  short-term  and  long-term 
stability  of  subtraction  errors,  and  found  that  only 
12Z  of  the  students  who  had  errors  on  the  first  test 
had  the  same  errora  on  the  second  (short  term)  test. 
He  also  found  chat  the  long  term  stability  data  are 
very  similar  to  the  short  term  stability  data.  VanLehn 
concluded  that  errors  in  general  are  not  stable. 
Further,  Brlcken  (1987)  investigated  the  stability  of 
algebra  errors,  and  found  that  50Z  (11  of  22  students) 
committed  at  least  one  error  on  the  pretest  which  reoc¬ 
curred  during  their  Interview  held  two  weeks  later. 
Although  the  studies  mentioned  above  all  confirm  that 
errors  generally  are  not  stable,  it  is  important  to 
note  that  each  investigation  measures  stability  using 
different  criteria.  However,  lack  of  stability  aeems 
to  be  an  issue  regardless  of  how  it  is  measured. 
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tially  concerned  to  know  the  Boet  comaonly  occurlng  (i.e-,  the  aost 
prevalent)  errora  In  the  data.  Whereas,  a  tutor  will  wish  to  have 
access  to  the  actual  error-profile  for  each  student . 


Taxonomy  of  errors:  This  study  has  confiraed  and  slightly  extended 
the  error  taxonoay  suggested  by  Sleeman  (1983).  In  our  view,  it  is 
feasible  to  talk  about  the  following  types  of  errors: 


Stable  errors  (both  reasdlable  and  "resistant") 


Attentions!  errors  (largely  minor  errors  such  as  adding/dropping 
signs) . 


Classes  of  aal-rules,  used  by  the  same  student  on  different  occa¬ 
sions  with  the  same  type  of  task. 


Mental  slips,  typing/transcription  errors. 


It  is  important  for  a  tutor  to  categorize  correctly  the  error  as, 
for  example,  it  aay  be  counterproductive  to  tutor  a  student  on  an 
error  which  is  a  result  of  a  slip;  whereas  it  aaybe  Important  to 
address  a  stable  error.  (These  Judgements  are  both  subtle  and  com¬ 
plex.) 


Overall  concluelons  of  the  eerlee  of  etudles 


For  introductory  algebra  when  taught  procedurally  with  this  type  of 
atudent,  it  appears  that  Sateachlng  la  as  effective  as  MBR.  From 
which  it  follows  that  CAl  would  be  as  effective  as  an  ITS  It  is 
vital  that  we  investigate  the  range  of  subjects,  instructional 
approaches  and  student  age-^ranges  for  which  this  result  holds. 
(This  Inforaation  will  be  laportant  for  both  educators  and  ITSs 
workers . ) 


Despite  the  conclusions  on  the  last  paragraph,  one  should  not  con¬ 
clude  that  reteachlng  by  a  classroon  teacher  will  be  as  effective 
as  reteachlng  in  a  one-to-one  tutoring  situation  or  by  a  computer 
system.  Immediacy  of  feedback  which  is  lacking  from  the  normal 
classroom  situation  may  well  be  a  critical  factor,  Lewis  &  Anderson 
(1985). 


*  It  is  critical  that  ITSs  receive  extensive  field-testing. 


*  The  subfield  of  ITS  should  not  conclude  that  the  task  of  building 
an  ITS  is  impossible,  but  it  should  conclude  that  the  task  is 
harder  than  we  had  initally  thought*,  c.f.,  Wlnograd  &  Flores 
(1986).  It  is  possible  that  the  more  global  analysis  (Moore  & 
Sleenan,  1987)  which  takes  account  of  the  student's  performance  at 
several  levels  might  be  needed  before  proper  remediation  can  be 
undertaken.  But  before  such  a  system  is  built  it  is  suggested  that 
experiments  are  run  to  see  whether  human  tutors  are  more  effective 
when  they  base  their  remediation  on  such  s  global  approach. 

*  Cronbach  and  Snow  (1977)  warned  that  producing  truly 
Individualised  instruction  was  a  demanding  task,  but 
the  ITS  field  chose  to  infer  that  their  conclusions 
were  unduly  pessimistic  given  the  complexity  of  analys¬ 
ing  such  a  diverse  set  of  experimental  studies. 


For  a  Bore  extensive  discussion  of  Che  conclusions  of  this  experlnent 
including  a  discussion  of  possible  additional  issues  to  be  investigated, 
see  Sleenan,  Kelly,  Martlnak,  Ward  and  Moore  (1967). 


Eeferences 


Bricken,  W.  M.  (1987).  Aaalytlng  Trorg  in  elementary  ■athematlcs ■ 
Doctoral  Dlaaertatlon,  School  of  Education,  Stanford  University. 


Buckingham,  G.  E.  (1933).  Diagnostic  and  remedial  teaching  in  first- 
year  algebra.  Northwestern  University,  IL,  School  of  Education  series. 
No.  11. 


Bunderson,  V.  C.  &  Olsen,  J.  B.  (1983).  Mental  errors  in  arithmetic 
skills;  their  diagnosis  in  pre-college  students .  WICAT  Education  Insti¬ 
tute  Provo,  UT.  Final  project  report,  NSF  SED  80-12500, 


Cronbach,  L.  J.  &  Snow,  R.  E.  (1977).  Aptitudes  and  Instructional 
Methods .  New  York:  Irvington. 


Hartley,  J,  R.  &  Sleeman,  D.  H.  (1973).  Towards  intelligent  teaching 
systems.  International  Journal  of  Man-Machine  Studies,  5,  215-236. 


Kelly,  A.  E.,  Martinak,  R.  &  Sleeman,  D.  (1987).  How  important  is  diag¬ 
nosis  for  remediation?  A  concern  for  intelligent  tutoring  systems. 
Proceedings  of  the  Second  European  Conference  for  Research  on  Learning 
and  Instruction,  Tubingen,  Federal  Republic  of  Germany,  September  19-27. 


Kelly,  A,  E.  &  Sleeman,  D.  (1986).  A  study  of  Diagnostic  and  Remedial 
Techniques  used  by  Master  Algebra  Teachers.  Technical  Report 
AUCS/TR8708,  Department  of  Computing  Science,  University  of  Aberdeen. 


64 


Lewis,  M.  W.  fc  Anderson,  J.  R.  (1985).  DlscrlBlnetlon  of  operator  sche- 
Bftta  in  problen  solving;  Learning  froa  exeaples.  Cognitive  Psychology, 
17_,  PP  26-65. 

Macnab,  D.  S.  &  CuasBlne,  J.  A.  (1986).  Teaching  Matheaatlcs  11-16;  A 
Difficulty-Centred  Approach.  London;  Basil  Blackwell,  England. 

Martlnak,  R. ,  Schneider,  B.  and  Sleeaan,  D.  (1987).  A  Coaparatlve 
Analysis  of  Approaches  for  Correcting  Algebra  Errors  via  an  Intelligent 
Tutoring  System.  Proceedings  of  AERA,  Washington  D.C. 

Moore,  J.  L.  &  Sleeman,  D.  (1987).  Enhancing  PIXIE'a  tutoring  capabili¬ 
ties.  Technical  Report  AUCS/TR8709,  Departaent  of  Coaputlng  Science  , 
University  of  Aberdeen. 

Putnam,  R>  T.  (1987).  Structuring  and  adjusting  Content  for  students:  A 
study  of  live  and  simulated  tutoring  of  addition.  Aaerlcan  Educational 
Research  Journal,  Vol  24,  No  1,  pp.  13  -  48 

Resnlck,  L.  (1984).  Beyond  error  analysis;  the  role  of  understanding  In 
elementary  school  arithmetic.  Learning  Research  and  Development  Centre, 
University  of  Pittsburgh,  PA. 

Resnlck  L.  B.  &  Ford  W.  W.,  (1981).  The  psychology  of  mathematics  for 
instruction.  Bllsdale,  NJ:  Lawrence  Erlbaum  Associates. 

Sleeasn,  D.  (1982).  Assessing  aspects  of  competence  In  basic  algebra. 
In  Intelligent  Tutoring  Systems  D.  Sleeaan  &  J.  S  Brown  (Eds).  London: 
Academic  Press,  pp  185-199. 

65 


SleeiuD,  D.  (1983).  Basic  algebra  revlalted:  a  study  with  14-year  olds. 
HPP  Report  83-9,  Conputer  Science  Dept,  Stanford  University.  Repub¬ 
lished  In  International  Journal  Man-Machine  Studies ,  (1985),  22,  pp 
127-149. 

Sleeman,  D.  (1987).  PIXIE:  A  shell  for  developing  Intelligent  Tutoring 
Systems.  In  R.W.  Lawler  4  M.  Tasdanl  (Eds).  Artificial  Intelligence 
and  Education:  Volume  1.  (pp  239-265).  Norwood,  N.J:  Ablex. 

Sleeman,  D.  &  Brown,  J.  S.  (1982).  Intelligent  Tutoring  Systems,  Lon¬ 
don:  Academic  Press. 


Sleeman,  D. ,  Kelly,  A.  E.,  Hartinak,  R.,  Ward,  R.  D.  6  Moore,  J.  (1987) 
Diagnosis  and  Remediation  In  the  context  of  Intelligent  Tutoring  Sys¬ 
tems.  Technical  Report  AUCS/TR8712,  Department  ot  Computing  Science, 
University  of  Aberdeen. 

Stelnburg,  R.  M. ,  Sleeman,  D.  and  Ktorza,  D.  (1985)  Transformations  and 
Equivalence  of  Equations  In  High  School  Algebra.  Technical  Report, 
School  of  Education,  Stanford. 

Swan,  M.  B.  (1983).  Teaching  decimal  place  value.  A  comparative  study 
of  conflict  and  positive  only  approaches.  Research  Report  31,  Shell 
Centre  for  Mathematical  Education,  University  of  Nottingham. 

VanLehn,  K.  (1981).  Bugs  are  not  enough:  Empirical  studies  of  bugs , 
impasses  and  repairs  In  procedural  skills.  XEROX,  Palo  Alto  Research 
Centre,  CA.  Cognitive  and  Instructional  Science  series,  ClS-11  (SSL- 


Pretest  Items 


Appendix  A 


Posttest  Items 


1. 

7x  -  14 

1. 

3x  -  9 

2. 

4x  ■  -12 

2. 

4x  -  -8 

3. 

5x  -  7 

3. 

3x  -  5 

4. 

CO 

1 

K 

00 

4. 

6x  -  9 

5. 

3x  -  8  +  4 

5. 

5x  -  8  +  2 

6. 

3x  -  4  *  A 

6. 

3x  -  4  *  3 

7. 

3x  +  4x  ■  14 

7. 

2x  +  3x  -  10 

8. 

3x  +  5  -  26 

8. 

2x  +  3  +  9 

9. 

7  +  4x  •  19 

9. 

2  +  4x  -  14 

10. 

5x  ■  4x  +  8 

10. 

4x  +  3x  +  6 

11. 

3x  -  3(2  +  3) 

11. 

2x  -  3(3  +  1) 

12. 

12x  -  2(3x  +  3) 

12. 

24x  -  3(2x  +  3) 

13. 

7x  +  2  +  4  *  8 

13. 

5x  -  2  +  3  *  8 

14. 

17x  -  19x  +  25 

14. 

16x  -  19x  +  20 

15. 

3  +  2x  +  4x  -  21 

15. 

2  +  2x  +  3x  -  17 

16. 

4  +  3x  +  4x  ■  25 

16. 

3  +  3x  +  4x  -24 

17. 

35x  -  2  +  3(4x  +  5) 

17. 

37x  -  2  +  3(4x  + 

Appendix  t  (Seaple  Scripts) 

Re  teach 


1.  [FRESH  PAFER] 

2.  Have  student  work.  Che  task  aloud 

3.  If  wrong,  say  "THIS  IS  WRONG". 

4.  [FRESH  PAPER] 

5.  Say,  "LET  ME  SHOW  YOU  HOW  lO  DO  IT  AND  WHY"  -  (using  th*  four 
rules) . 

6.  GIVE  PRACTICE  TASKS 

FOUR  RULES; 

1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  "Get  all  the  Xa  to  one  side,  all  the  nuabers  to  the  other" 

3.  "To  undo  added  things,  you  subtract, 

to  undo  aultlplled  things,  you  divide." 

4.  "Whatever  you  do  to  one  side,  you  Bust  do  the  saae  thing  to  the 


other  side." 

i 


1.  I FRESH  PAPER] 


2.  Have  atudent  work  the  task  aloud 


3.  After  the  atudent  has  completed  the  task,  go  back  to  EACH  error. 


"IT  LOOKS  LIKE  YOU  (DID)  . THIS  IS  WRONG  BECAUSE 


(Address  the  Four  Rules) 


4.  Say,  "LET  ME  SHOW  YOU  TO  DO  IT  AND  WHY”  -  (Using  the  four 
rules). 


5.  GIVE  PRACTICE  TASKS 


FOUR  RULES 


1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 


2.  "Get  all  the  Xs  to  one  side,  all  the  numbers  to  the  other' 


3.  "To  undo  added  things,  you  subtract, 

to  undo  multiplied  things,  you  divide. 


4.  "Whatever  you  do  to  one  side,  you  must  do  the  same  thing  to  the 
other  side." 


1.  [FRESH  PAPER] 


2«  Have  student  work  the  task  aloud 

3.  After  the  student  has  completed  the  task,  go  back  to  EACH  error, 

say: 

"IT  LOOKS  LIKE  YOU  (DID)  . THIS  IS  WRONG  BECAUSE  ..." 

(Address  the  Four  Rules) 

4.  Say,  "LET  ME  SHOW  YOU  HOW  TO  DO  IT  AND  WHY"  -  (Using  the  four 
rules). 

5.  GIVE  PRACTICE  TASKS 

FOUR  RULES 

1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  "Get  all  the  Xs  to  one  side,  all  the  numbers  to  the  other" 

3.  "To  undo  added  things,  you  subtract, 

to  undo  multiplied  things,  you  divide." 

4.  "Whatever  you  do  to  one  side,  you  must  do  the  same  thing  to  the 


other  side 


cirai 


Appendix  C 


ITDIS  TO  USE  FOR  TUTORING  (the  first  Itea  is  from  the  pretest) 


fH 

1 

M 

2. 

C4 

1 

1 

M 

8x  -  16 

6x  -  -24 

3x  -  6 

5x  -  -15 

5x  -  7 

4. 

8x  -  18 

6x  -  13 

9x  -  15 

4x  -  9 

1 

H 

3x  -  8  +  4 

6. 

3x  ■  4  *  4 

5x  -  19  +  6 

2x  -  3  *  3 

2x  ■  7  +  5 

8x  -  5  *  2 

3x  +  4x  •  1 

8. 

3x  +  5  -  26 

6x  +  3x  ■  36 

3x  +  4  -  19 

2x  +  3x  -  20 

6x  +  3  -  21 

7  +  4x  -  19 

10. 

5x  ■  4x  +  8 

6  +  4x  ■■  26 

7x  -  5x  +  18 

3  +  5x  -  11 

6x  ■  3x  +  2 

3x  -  3(2  +  3) 

12. 

12x  -  2(3x  +  3) 

4x  -  3(4  +  5) 

I7x  -  2(4x  +  3) 

5x  -  4(1  +  4) 

14x  -  3(3x  +  7) 

7x  -  2  +  4  *  8 

14. 

17x  -  19x  +  25 

2x  -  2  +  4  *  6 

I5x  •  18x  +  17 

5x  -  8  +  3  *  4 

21x  -  24x  +  13 

3  +  2x  +  4x  -  21 

16. 

4  +  3x  +  4x  ■  25 

• -  ‘ A  .N^NVr  •nV.X.V.-s.'-.'l''."*'.'  -v"'.”.-.'  o  v'  o 
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Appendix  _C 


ITEMS  TO  USE  FOR  TUTORING  (the  first  Itea  is  from  the  pretest) 


1. 

■ 

H 

2. 

4x  -  -12 

8x  ■  16 

9» 

H 

■ 

I 

3x  ■  6 

5x  -  -15 

3. 

5x  -  7 

4. 

8x  -  18 

6x  -  13 

9x  -  15 

4x  -  9 

6x  -  14 

5. 

3x  -  8  +  4 

6. 

3x  -  4  *  4 

5x  -  19  +  6 

2x  -  3  *  3 

2x  -  7  +  5 

8x  -  5  *  2 

7. 

3x  +  4x  -  1 

8. 

3x  +  5  -  26 

6x  +  3x  ■  36 

3x  +  4  -  19 

2x  +  3x  •  20 

6x  +  3  -  21 

9. 

7  +  4x  -  19 

10. 

•^r  -  4x  +  8 

6  +  4x  ■  26 

7x  -  5x  +  18 

3  +  5x  -  11 

6x  •  3x  +  2 

11. 

3x  -  3(2  +  3) 

12. 

12x  -  2(3x  +  3) 

4x  -  3(4  +  5) 

17x  -  2(4x  +  3) 

5x  -  4(1  +  4) 

14x  -  3(3x  +  7) 

13. 

7x  -  2  +  4  *  8 

14. 

I7x  -  19x  +  25 

2x  -  2  +  4  *  6 

15x  -  18x  +  17 

5x  -  8  +  3  *  4 

21x  -  24x  +  13 

15. 

3  +  2x  +  4x  -  21 

16. 

4  +  3x  +  4x  ■  25 
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Retcach 


1.  [FRESH  PAPER] 


2.  Have  atudent  work  the  teak  aloud 


3.  If  %nroag,  aay  "THIS  IS  WRONG**. 


4.  [FRESH  PAPER] 


5.  Say,  **LET  ME  SHOW  YOU  HOW  TO  DO  IT  AND  WHY"  -  (using  the  four 
rules) . 


6.  GIVE  PRACTICE  TASKS 


FOUR  RULES: 


1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  "Get  all  the  Xa  to  one  aide,  all  the  numbers  to  the  other" 

3.  "To  undo  added  things,  you  subtract, 

,to  undo  multiplied  things,  you  divide." 


Reteach 


1.  [FRESH  PAPER] 

2.  Have  student  work  the  Cask  aloud 

3.  If  wrong,  aay  “THIS  IS  WRONG". 

4.  [FRESH  PAPER] 

5.  Say,  "LF;  ME  SHOW  YOU  ^  TO  DO  IT  AND  WHY"  -  (using  the  four 
rules) . 

6.  GIVE  PRACTICE  TASKS 

POUR  RULES; 

1.  Precedence  -  “My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  “Get  all  the  Xs  to  one  side,  all  the  numbers  to  the  other" 

3.  “To  undo  added  things,  you  subtract, 

to  undo  multiplied  things,  you  divide." 

4.  “Whatever  you  do  to  one  side,  you  must  do  the  same  thing  to  the 
other  side." 
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1.  [FRESH  PAPER] 


KBR 


2.  HAve  student  work  the  teak  aloud 

3.  After  the  student  has  conpleted  the  task,  go  back  to  EACH  error, 
say : 

"IT  LOOKS  LIKE  YOU  (DID)  . THIS  IS  WRONG  BECAUSE  ..." 

(Address  the  Four  Rules) 

4.  Say,  "LET  ME  SHOW  YOU  HOW  TO  DO  IT  AND  WHY"  -  (Using  the  four 
rules) . 

5.  GIVE  PRACTICE  TASKS 

FOUR  RULES 

1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  befrre 

Adding  or  Subtracting 

2.  "Get  all  the  Xs  to  one  side,  all  the  nuabers  to  the  other" 

3.  "To  undo  added  things,  you  subtract, 

to  undo  aultlplled  things,  you  divide." 

4.  "Whatever  you  do  to  one  side,  you  must  do  the  saae  thing  to  the 


other  side. 


1.  [FRESH  PAPER] 


2.  Save  atudent  work  the  task  aloud 

3.  After  the  atudeat  has  completed  the  task,  go  back  to  EACH  error, 
aay : 

“IT  LOOKS  LIKE  YOU  (DID)  . THIS  IS  WRONG  BECAUSE  ..." 

(Address  the  Four  Rules) 

4.  Say,  “LET  KE  SHOW  YOU  ^  TO  D^)  IT  AND  WHY"  -  (Using  the  four 
rules) . 

5.  GIVE  PRACTICE  TASKS 


FOUR  RULES 

1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  “Get  all  the  Xs  to  one  aide,  all  the  numbers  to  the  other" 

3.  “To  undo  added  things,  you  subtract, 

to  undo  multiplied  things,  you  divide." 

4.  “Whatever  you  do  to  one  aide,  you  must  do  the  tame  thing  to  the 

other  side." 


''v 
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HBR  +  "cognitive  engagement " 


[FRESH  PAPER] 

Have  the  student  work  the  teak  aloud 

“TELL  ME  HOW  (AND  WHY)  YOU  DID  (the  errors)"  No  need  to  review 
correct  etepa.  (Student  "targets"  own  errors] 

For  each  error  say,  "THIS  IS  WRONG." 

Say,  "LET  ME  SHOW  YOU  HOW  TO  DO  IT  AFH)  W:'  -  (Using  the  four 

rules) . 

[FRESH  PAPER] 

Re-present  task. 

Say,  "NOW  YOU  TELL  ME  HOW/WHY  TO  DO  THIS  TASK" 

You  need  a  how  and  why  for  each  step,  if  possible.  If  (s)he  makes 
an  error,  correct  it  on  the  spot;  (s)he  does  not  have  to  repeat 
this  step. 


GIVE  PRACTICE  TASKS 


M  in  in 
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MBR  +  "cognitive  engagement" 


1.  (FRESH  PAPER] 

2.  Have  the  student  work  the  task  aloud 

3.  "TELL  ME  HOW  (AND  WHY)  YOU  DID  (the  errors)"  No  need  to  review 
correct  steps.  [Student  "targets"  own  errors] 

4.  For  each  error  say,  "THIS  IS  WRONG." 

5.  Say,  "LET  ME  SHOW  YOU  nw  TO  DO  IT  AND  ’  -  (Uslr.  the  four 

rules) . 

6.  (FRESH  PAPER] 

7.  Re*preeent  task. 

Say ,  "NOW  YOU  TELL  ME  HOW/WHY  TO  DO  THIS  TASK" 

You  need  a  how  and  why  for  each  step,  if  possible.  If  (s)he  Bakes 
an  error,  correct  It  on  the  spot;  (s)he  does  not  have  to  repeat 
this  step. 


8.  GIVE  PRACTICE  TASKS 


FOUR  RUUS 


1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Su  ^ractlng 

2.  "Get  all  the  Xa  to  one  aide,  all  the  nuabera  to  the  other" 

3.  "To  undo  added  thlnga,  you  aubtract, 

to  undo  Bultlplled  thlnga,  you  divide-" 

4.  "Whatever  you  do  to  one  aide,  you  Bust  do  the  aame  thing  to  the 

other  aide." 
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FOUR  RULES 


1.  Precedence  -  "My  Dear  Aunt  Sally"  *■  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  "Get  all  the  Xa  to  one  aide,  all  the  nuabers  to  the  other" 

3.  "To  undo  added  things,  you  aubtract, 

to  undo  aultlplled  things,  you  divide." 

4.  "Whatever  you  do  to  one  aide,  you  Bust  do  the  ease  thing  to  the 


other  side." 


MBR  +  "cognitive  dlaionence 


i  [FRESH  PAPER] 

2.  Have  the  student  work  the  task  aloud 

3.  Even  If  the  answer  la  correct,  say, 

"PLEASE  CHECK  YOUR  ANSWER" 

Have  the  student  aubstltute  their  answer  for  X.  If  they  don't 
remember  substitution,  remind  them.  Substitute  in  their  wrong 
answer,  and  have  them  agree  that  the  two  sides  do  not  balance.  Ask, 

"HOW  DO  WE  KNOW  THAT  IT  IS  WRONG?"  -  because  the  two  sides  don't 

balance.  If  they  seem  TOTALLY  lost,  you  should  give  a  very  obvious 
example  (e.g.,  5X-15). 

If  their  wrong  answer  DOES  balance  the  sides,  say,  "EVEN  THOUGH 

THIS  VALUE  FOR  X  IS  RIGHT,  YOU  COT  IT  FOR  THE  WRONG  REASON . (and 

explain) . . . 

4.  Say,  "THIS  IS  WRONG"/"THIS  METHOD  IS  WRONG." 

5.  Say,  "LET  ME  SHOW  YOU  HOW  TO  DO  IT"  (And  WHY  -  Using  the  four 
rules}"  Do  entire  task. 

6.  GIVE  PRACTICE  TASKS 
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FOUR  RULES 


1.  Precedence  -  "My  Dear  Aunt  Sally**  -  Multiply  or  Divide  before 

Adding  or  Subtracting 

2.  **601  all  the  Xs  to  one  aide,  all  the  nuabers  to  the  other** 

3.  "To  undo  added  things,  you  subtract, 

to  undo  aultlplled  things,  you  divide.** 

4.  "Whatever  you  do  to  one  aide,  you  Bust  do  the  same  thing  to  the 

other  aide.** 


FOUR  RULES 


1.  Precedence  -  "My  Dear  Aunt  Sally"  -  Multiply  or  Divide  before 

Adding  or  Subtracting 


2.  "Get  all  the  Xa  to  one  aide,  all  the  nuabers  to  the  other* 


3.  "To  undo  added  things,  you  subtract, 

to  undo  multiplied  things,  you  divide.' 


4.  "Whatever  you  do  to  one  aide,  you  must  do  the  same  thing  to  the 
other  aide." 


Appendix  E  (Cleeglf Icetlon  of  errors) 


Algebraic  Errors 


1.  Changed  the  aide  but  not  algn  of  an  x-term 


2.  Changed  the  aide  of  the  larger  positive  x-tenn  but  placed  the  alnus 
algn  before  the  aaaller  x*tenn. 


3.  Added  negative  algn(B),  a.g.,  3r+5-9+6  ■>  3x— 9-6-5 


4.  Added  an  x-tera  to  a  constant «  e.g.,  4x+3  ■>  7x 


5.  Ended  with  the  step  ax~bx. 


6.  Inverted  division.  ax“b  ■>  x*a/b 


7.  Subtracted  the  coefficient. 


B.  Changed  the  aide  of  a  aultlpller, 

e.g.,  5*3x+7-9->3x+7-9*5 


9.  Dropped  an  x,  e.g.,  3x+3"6x  ■>  3x+3-6 


10.  Other  low  frequency  algebraic  errors. 


Non  algebraic  errors 


Non  algebraic  errors 


1.  Precedence  errors 

2.  Distributive  property  errors 

3.  Klsreading  a  aign/number 

4.  Incoaplete  working 

5.  Item  not  attempted 

6.  Arithmetic  errors 

7.  Dropping  a  negati've  sign 

6.  Other  non-algebralc  errors. 


sndix  F 


PRETEST 


Teacher: 


'i}ate  of  Birth: 

Today's  Date: 


IKSTRUCTIONS :  Please  solve  for  x.  Please  show  all  of  the  steps  you 
take  for  each  task.  Use  the  hack  of  the  paper  if  you  need  aore  space. 


1.  3x  -  9 

2.  2x 

o 

1 

3.  9x 

-  18 

4.  3x  ■  -9 

5.  2x 

-  -10 

6.  9x 

-  -18 

X 

1 

8.  7x 

-  2 

9.  8x 

-  5 

10.  5x  -  -3 

11.  7x 

m  -2 

12.  8x 

-  -5 

13.  3x  -  5 

14.  7x 

-  9 

15.  5x 

-  7 

16.  3x  -  -5 

17.  7x 

•  -9 

18.  5x 

-  -7 

19.  4  +  3x  -  19 

20. 

5  +  2x  -  20 

21. 

6  +  4x  -  21 

22.  9x  +  5x  ■  70 

23. 

6x  +  4x  ■  20 

24. 

8x  +  5x  -  26 

25.  3x  +  4  -  19 

26. 

2x  +  8  -  21 

27. 

7x  +  9  -  13 

28.  7x  -  5x  +  17 

29. 

12x  -  9x  +  21 

30. 

lOx  -  6x  +  19 

31.  2x  -  3{3  +  1) 

32. 

4x  -  3(2  +  5) 

33. 

6x  -  3(4  +  2) 

34.  7x  -  2(5x  +  6) 

35. 

6x  ■  4(3x  +  4) 

36.  3x  -  3(5x  +  6) 

37. 

2x  -  2  +  4  *  6 

38.  4x  -  12  +  8  *  2 

39. 

5x  -  6  +  2  *  3 
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Appendix  £ 


PRETEST 


KAME:  _  Date  of  Birth:  _ 

Teacher:  Today's  Date:  _ 

INSTRUCTIONS :  Please  aolve  for  x.  Please  show  all  of  the  steps  you 


take  for  each  task. 

Use  the  back  of  the  paper 

if  you  need  sore  apace 

1.  3x  -  9 

2.  2x  -  10 

3.  9x  18 

4.  3x  ■  -9 

5.  2x  -  >10 

6.  9x  -  -18 

7.  5x  -  3 

8.  7x  -  2 

9.  8x  5 

10.  5x  -  -3 

11.  7x  -  -2 

12.  8x  -  -5 

13.  3x  -  5 

14.  7x  -  9 

15.  5x  -  7 

16.  3x  «  -5 

17.  7x  -  -9 

18.  5x  -  -7 

19.  4  +  3x  -  19 

20.  5  +  2x  -  20 

21.  6  +  4x  -  21 

22.  9x  +  5x  -  70 

23.  6x  +  4x  -  20 

24.  8x  5x  ■  26 

25.  3x  +  4  -  19 

26.  2x  8  -  21 

27.  7x  +  9  -  13 

28.  7x  -  5x  +  17 

29.  12x  -  9x  21 

30.  lOx  -  6x  +  19 

31.  2x  -  3(3  +  1) 

32.  4x  -  3(2  +  5) 

33.  6x  -  3(4  +  2) 

34.  7x  -  2(5x  +  6) 

35.  8x  -  4(3x  +  4) 

36.  3x  ■  3(5x  +  6) 

37.  2x  -  2  4  •  6 

38.  4x  -  12  +  8  *  2 

39.  5x  -  6  +  2  *  3 
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40.  2  +  3x  +  4x  -  16 

42.  7  +  9x  +  4x  -  24 
44.  30x  -  2  +  7(2x  +  6) 
46.  2  *  3x  +  2x  -  12 

48.  7  *  2x  +  6x  -  13 
50.  22x  -  5  •  2(2x  +  3) 


41.  4  5x  2x  -  25 
43.  36x  -  2  +  3(4x  +  5) 
45.  31x  -  4  +  2(4x  +  6) 
47.  4  *  3x  +  4x  ■  7 
49.  20x  -  3  *  3(2x  +  4) 
51.  31x  -  4  *  2(2x  +  3) 


40.  2  + 

3x  +  4x  -  16 

41.  4  + 

42.  7  + 

9x  +  4x  ■  24 

43.  36x 

44.  30x 

-  2  +  7(2x  +  6) 

45.  31x 

46.  2  * 

3x  +  2x  -  12 

47.  4  • 

48.  7  * 

2x  +  6x  -  13 

49.  20x 

50.  22x 

-  5  •  2(2x  +  3) 

51.  31x 

W!W?IWWW 


POSTTEST 


NAME: 

Date  of  Birth: 

Teacher: 

Today'a  Date: 

INSTRUCTIONS: 

Pleaae  aolve  for  x.  Please  ahow  all  of 

the  steps  you 

take  for  each  task.  Uae  the  back  of  the  paper  if  you  need  aore  apace. 
1.  3x  -  9  2.  7x  •  14  3.  8x  -  16 


4.  3x  -  -9 

5.  7x  -  -14 

6.  8x  ■ 

-16 

7.  5x  -  3 

8.  7x  -  5 

9.  3x  - 

2 

10.  5x  -  -3 

11.  7x  -  -5 

12.  3x  - 

-2 

13.  3x  -  5 

14.  8x  -  11 

15.  4x  - 

7 

16.  3x  ■  -5 

17.  8x  -  -11 

18.  4x  ■ 

-7 

19.  4  +  3x  -  19 

20.  5  +  3x  -  21 

21.  6  +  7x  •  24 

22.  9x  +  5x  -  70 

23.  2x  +  8x  -  40 

24.  3x  + 

7x  ■  60 

25.  3x  +  4  -  19 

26.  6x  +  4  •  15 

27.  7x  + 

2-6 

28.  7x  -  5x  +  17 

29.  9x  -  6x  +  12 

30.  12x  -  6x  +  23 

31.  2x  -  3(3  +  1) 

32.  3x  -  2(3  +  4) 

33.  7x  - 

5(4  +  3) 

34.  7x  -  2(5x  +  6) 

35.  2x  -  4(3x  +  5) 

36.  4x  “  6(ix  +  8) 

37.  2x  -  2  +  4  *  6 

38.  4x  -  12  +  3  •  4 

39.  6x  -  3  +  3  *  2 

40.  2  ♦  3x  +  4x  -  16 

41.  3  +  6x  +  2x  ■  19 
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POSTTEST 


NAME:  Date  of  Birth: 

Teacher:  _  Today'a  Date:  _ 

INSTRUCTIONS :  Pleaae  aolve  for  3t .  Pleaae  ahow  all  of  the  steps  you 
take  for  each  task.  Use  the  back  of  the  paper  If  you  need  sore  apace. 


1.  3x  -  9  2. 

7x  •  14 

3.  8x  -  16 

4.  3x  -  -9 

5.  7x  -  -14 

6.  8x  ■  -16 

7.  5x  -  3 

8.  7x  -  5 

9.  3x  -  2 

10.  5x  -  -3 

11.  7x  -  -5 

12.  3x  -  -2 

13.  3x  -  5 

14.  8x  -  11 

15.  4x  -  7 

16.  3x  -  -5 

17.  8x  -  -11 

18.  4x  -  -7 

19.  4  +  3x  -  19 

20.  5  +  3x  -  21 

21.  6  +  7x  -  24 

22.  9x  +  5x  -  70 

23.  2x  +  8x  -  40 

24.  3x  -f  7x  ■  60 

25.  3x  +  4  -  19 

26.  6x  +  4  ■  15 

27.  7x  +  2  -  6 

28.  7x  -  5x  +  17 

29.  9x  -  6x  12 

30.  12x  -  6x  +  23 

31.  2x  -  3(3  +  1) 

32.  3x  -  2(3  +  4) 

33.  7x  -  5(4  +  3) 

34.  7x  -  2(5x  +  6) 

35.  2x  -  4(3x  +  5) 

36.  4x  ■  6(5x  +  8) 

37.  2x  -  2  +  4  *  6 

38.  4x  -  12  +  3  *  4 

39.  6x  -  3  3  < 

‘  2 

40.  2  3x  4x  -  16 

41.  3  +  6x  +  2x 

-  19 
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42.  4  +  2x  +  7x  •  24 


43.  36x  -  2  +  3(4x  +  5) 


44.  38x  -  3  +  6(3x  +  5)  45.  31x  -  3  +  5(2x  +  3) 

46.  2  *  3x  +  2x  -  12  47.  2  •  7x  +  2x  •  7 

48.  5  *  3x  +  4x  -  11  49.  20x  -  3  •  3(2x  +  4) 

50.  23x  -  3  *  3(2x  +  5)  51.  35x  -  2  *  5(2x  +  4) 


